Configure a Disaster Recovery System

This topic describes how to configure your  nodes for Disaster Recovery.
gateway83
This topic describes how to configure your 
API Gateway
 nodes for Disaster Recovery.
Configuring a Disaster Recovery System requires advanced knowledge and is intended for system administrators or other technical users.
In this topic, the two nodes in the cluster are referred to as 
Gateway1
 and 
Gateway2,
where
 Gateway1
is the database node hosting the primary database. The DR node being configured is referred to as
Gateway-DR.
Contents:
2
2
What Scripts Do I Need?
Use the following scripts to configure disaster recovery:
  • add_slave_user.sh
  • create_DR_slave.sh
  • monitor_replication.sh
Preparing the Secondary DB node on Gateway2
Required Script:
add_slave_user.sh
  1. Access the privileged shell on Gateway2.
  2. Run
    /opt/SecureSpan/Appliance/bin/add_slave_user.sh
    to add a new Access Control List (ACL) to the MySQL database for the DR system to replicate. 
  3. Complete the prompts displayed on the screen
    For more information, see Configuring Cluster Database Replication.
#
/opt/SecureSpan/Appliance/bin/add_slave_user.sh -v
Gathering information for SLAVE user
Enter hostname or IP for the SLAVE: 
Gateway-DR.l7tech.com
Enter replication user: [repluser]
repluser
Enter replication password: [replpass]
replpass
Enter MySQL root user: [root]
root
Enter MySQL root password: []
rootpass
Checking configuration of running MySQL...
MySQL appears to be properly configured with server_id=2
Do you want to continue? [Y]
Y
Granting slave permissions to repluser@Gateway-DR.l7tech.com
Done.
Configuring the DR Node Replication
Required Script: 
create_DR_slave.sh
  1. Access the privileged shell on the Gateway-DR.
  2. Edit
    /etc/my.cnf
    and add a line to set
    server-id=3
    :
    # Uncommment log-bin and log-slave-update if a clustered
    # db server
    #log-bin=/var/lib/mysql/ssgbin-log
    #log-slave-update
    # uncomment the next item on 1st db master server
    #server-id=1
    # uncomment the next item on 2nd db master servers
    #server-id=2
    # This is a DR node 
    server-id=3
    relay-log = /var/lib/mysql/ssgrelay-bin
    relay-log-index = /var/lib/mysql/ssgrelay-bin.index
    relay-log-info-file = /var/lib/mysql/ssgrelay-bin.info
  3. Restart the mysqld service:
    #
    service mysqld restart
    Stopping MySQL: [ OK ]
    Starting MySQL: [ OK ]
  4. Use
    scp
    to copy the
    create_DR_slave.sh
    script to the home directory of the
    ssgconfig
    user. 
  5. Run the following comments to set the ownership and permissions for the script file, and then move it to
    /usr/local/bin
    :
     
    #
    chown root:root ~ssgconfig/create_DR_slave.sh
    #
    chmod 700 ~ssgconfig/create_DR_slave.sh
    #
    mv ~ssgconfig/create_DR_slave.sh /usr/local/bin
  6. Run the
    create_DR_slave.sh
    script to configure the slave database:
    #
    /usr/local/bin/create_DR_slave.sh -v
    Enter hostname or IP for the secondary DB node in the cluster: 
    Gateway2.mycompany.com
    Enter replication user: [repluser]
    repluser
    Enter replication password: [replpass]
    replpass
    Enter monitor user: [monitor]
    monitor
     
    Enter monitor password: [monitorpass]
    monitorpass
    Enter MySQL root user: [root]
    root
    Enter MySQL root password: []
    7layer
    Do you want to clone a database from Gateway2.l7tech.com (yes or no)? [yes]
    yes
    Enter name of database to clone: [ssg]
    ssg
    --> MASTER = Gateway2.l7tech.com
    --> DBUSER = repluser
    --> DBPWD = replpass
    --> ROOT = root
    --> ROOT_PWD = 7layer
    --> CLONE_DB = yes
    --> DB = ssg
    --> Stopping slave
    --> File = ssgbin-log.000002
    --> Position = 62095412
    --> Changing MASTER settings
    --> Creating database: ssg
    --> Copying database from Gateway2.l7tech.com
    --> Starting slave
    --> Confirming slave startup
    --> Slave_IO_Running = Yes
    --> Slave_SQL_Running = Yes
    Slave successfully created
    --> Getting password for user gateway 
    --> Granting access for 'gateway'@'localhost' 
    --> Granting access for 'gateway'@'%' 
    --> Granting access for 'gateway'@'localhost.localdomain' 
    --> Granting monitor rights for monitor@localhost 
    Does the clone command time out? If so, see Troubleshooting
  7. Exit the privileged shell to the Gateway main menu.
    Select option
    2
    (Display CA API Gateway configuration menu) and then option
    3
    (Configure the CA API Gateway).
  8. Configure the DR Gateway to use the localhost database but remain in a disabled state:
  9. ---------------------------------------------------------------------- 
    Configure CA API Gateway 
    ---------------------------------------------------------------------- 
    At any time type "quit" to quit.
    ----------------------- 
    Set Up the CA API Gateway Database 
    ----------------------- 
    At any time type "quit" to quit. 
    Press "<" to go to the previous step.
    This step lets you create or set up a connection to the SSG database.
    Enter the database hostname.
    Database Host [localhost]:
    localhost
    Enter the database port.
    Database Port [3306]:
    3306
    Enter the database name.
    Database Name [ssg]:
    ssg
    Enter the database user.
    Database Username [gateway]:
    gateway
    Enter the database password.
    Database Password:
    db_password
    Confirm Database Password:
    db_password
    -------------------------------- 
    Set Up the CA API Gateway Failover Database 
    -------------------------------- 
    At any time type "quit" to quit. 
    Press "<" to go to the previous step.
    This step lets you create or set up a connection to the SSG failover 
    database.
    This step is optional, enter "Yes" to continue or "No" to skip.
    Configure Database Failover Connection? [No]:
    No
    ---------------------- 
    Set Up the CA API Gateway Cluster 
    ---------------------- 
    At any time type "quit" to quit. 
    Press "<" to go to the previous step.
    This step lets you set up the SSG cluster.
    Enter the cluster passphrase (6-128 characters).
    Cluster Passphrase:
    cluster_passphrase
    Confirm Cluster Passphrase:
    cluster_passphrase
    ------------------- 
    Set Up the SSG Node 
    ------------------- 
    At any time type "quit" to quit. 
    Press "<" to go to the previous step.
    This step lets you set up the SSG node.
    Enable or disable the node.
    Enabled [Yes]:
    No
    --------------------- 
    Configuration Summary 
    --------------------- 
    Press < to go to the previous step, type "quit" to quit.
    The following configuration will be applied:
    Database Connection 
    Database Host = localhost 
    Database Port = 3306 
    Database Name = ssg 
    Database Username = gateway
    Node Configuration 
    Enabled = No
    Press [Enter] to continue. 
    Please wait while the configuration is applied ... 
    --------------------- 
    Configuration Results 
    ---------------------
    The configuration was successfully applied.
    Press [Enter] to continue
Monitoring the Replication State
Required Script: 
monitor_replication.sh
  1. Use
    scp
    to copy the
    monitor_replication.sh
    script to the home directory of the
    ssgconfig
    user. 
  2. Open a privileged shell on Gateway-DR.
  3. Run the following commands to change the ownership and permissions of the 
    monitor_replication.sh
     script to the home directory of the ssgconfig user: 
    #
    chown root:root ~ssgconfig/monitor_replication.sh
    #
    chmod 700 ~ssgconfig/monitor_replication.sh
    #
    mv ~ssgconfig/monitor_replication.sh /usr/local/bin
  4. Open
    /usr/local/bin/monitor_replication.sh
    for editing and set the configurable settings, including:
    Setting
    Notes
    MONUSER
    Must be same as value in
    create_DR_slave.sh
    MONPWD
    Must be same as value in
    create_DR_slave.sh
    NOTIFY
    Use default unless you have a specific setting in mind
    NOTIFY_SMTP
    Configure
    exim
    as per notes in script
    → NOTIFY_TO
    Required
    → NOTIFY_CC
    Optional
    → NOTIFY_BCC
    Optional
    → MTA_FLAG
    See notes for value; use default unless you have a specific setting in mind
    NOTIFY_SNMP
    If you use SNMP for notification, ensure that
    net-snmp-utils
    is installed. For more information, see the notes within the script.
    → SNMP_HOST
    Required
    → COMMUNITY
    Required
    → OID_ERROR
    Use default unless you have a specific setting in mind
    VERBOSE
    Set to 'yes' for testing and 'no' for production
  5. Test the
    monitor_replication.sh
    script file:
    #
    monitor_replication.sh
    Master_Host = Gateway2.l7tech.com
    Slave_IO_Running = Yes
    Slave_SQL_Running = Yes
    Master_Log_File = ssgbin-log.000005
    Slave is functioning properly.
    #
    mysqladmin stop-slave
    Slave stopped
    #
    monitor_replication.sh
    Master_Host = Gateway2.l7tech.com
    Slave_IO_Running = No
    Slave_SQL_Running = No
    Master_Log_File = ssgbin-log.000005
    WARNING: Slave is not functioning properly
    --- This *should* have been sent by SMTP or SNMP ---
    Sending notification by SNMP trap
    #
    mysqladmin start-slave
    Slave started
    #
    monitor_replication.sh
    Master_Host = Gateway2.l7tech.com
    Slave_IO_Running = Yes
    Slave_SQL_Running = Yes
    Master_Log_File = ssgbin-log.000005
    Slave is functioning properly.
  6. In a production environment, edit
    /usr/local/bin/monitor_replication.sh
    and set
    VERBOSE='no'
  7. Add
    /usr/local/bin/monitor_replication.sh
    to crontab. For more information, see the notes within the script for more information.
    #
    crontab -e
    # Monitor DR database replication
    0 * * * * /usr/local/bin/monitor_replication.sh
Troubleshooting
Resolving Clone Timeouts
If the database is very large, the clone command may time out. If this happens, access a privileged shell on Gateway2 and manually dump the database with:
# mysqldump --master-data=1 -r ssgdump.sql ssg
Then copy 
ssgdump.sql
 to the Gateway-DR and load it:
# mysqladmin stop-slave
# mysqladmin create ssg
# mysql ssg < ssgdump.sql
# mysqladmin start-slave
# mysql -e "SHOW SLAVE STATUS\G" | grep Running 
Slave_IO_Running: Yes 
Slave_SQL_Running: Yes