Configure a Disaster Recovery System
This topic describes how to configure your nodes for Disaster Recovery.
gateway83
This topic describes how to configure your
API Gateway
nodes for Disaster Recovery. Configuring a Disaster Recovery System requires advanced knowledge and is intended for system administrators or other technical users.
In this topic, the two nodes in the cluster are referred to as
Gateway1
and Gateway2,
where Gateway1
is the database node hosting the primary database. The DR node being configured is referred to as Gateway-DR.
Contents:
2
2
What Scripts Do I Need?
Use the following scripts to configure disaster recovery:
- add_slave_user.sh
- create_DR_slave.sh
- monitor_replication.sh
Preparing the Secondary DB node on Gateway2
Required Script:
add_slave_user.sh
- Access the privileged shell on Gateway2.
- Run/opt/SecureSpan/Appliance/bin/add_slave_user.shto add a new Access Control List (ACL) to the MySQL database for the DR system to replicate.
- Complete the prompts displayed on the screenFor more information, see Configuring Cluster Database Replication.
#/opt/SecureSpan/Appliance/bin/add_slave_user.sh -vGathering information for SLAVE userEnter hostname or IP for the SLAVE:Gateway-DR.l7tech.comEnter replication user: [repluser]repluserEnter replication password: [replpass]replpassEnter MySQL root user: [root]rootEnter MySQL root password: []rootpassChecking configuration of running MySQL...MySQL appears to be properly configured with server_id=2Do you want to continue? [Y]YGranting slave permissions to repluser@Gateway-DR.l7tech.comDone.
Configuring the DR Node Replication
Required Script:
create_DR_slave.sh
- Access the privileged shell on the Gateway-DR.
- Edit/etc/my.cnfand add a line to setserver-id=3:# Uncommment log-bin and log-slave-update if a clustered# db server#log-bin=/var/lib/mysql/ssgbin-log#log-slave-update# uncomment the next item on 1st db master server#server-id=1# uncomment the next item on 2nd db master servers#server-id=2# This is a DR nodeserver-id=3relay-log = /var/lib/mysql/ssgrelay-binrelay-log-index = /var/lib/mysql/ssgrelay-bin.indexrelay-log-info-file = /var/lib/mysql/ssgrelay-bin.info
- Restart the mysqld service:#service mysqld restartStopping MySQL: [ OK ]Starting MySQL: [ OK ]
- Usescpto copy thecreate_DR_slave.shscript to the home directory of thessgconfiguser.
- Run the following comments to set the ownership and permissions for the script file, and then move it to/usr/local/bin:#chown root:root ~ssgconfig/create_DR_slave.sh#chmod 700 ~ssgconfig/create_DR_slave.sh#mv ~ssgconfig/create_DR_slave.sh /usr/local/bin
- Run thecreate_DR_slave.shscript to configure the slave database:#/usr/local/bin/create_DR_slave.sh -vEnter hostname or IP for the secondary DB node in the cluster:Gateway2.mycompany.comEnter replication user: [repluser]repluserEnter replication password: [replpass]replpassEnter monitor user: [monitor]monitorEnter monitor password: [monitorpass]monitorpassEnter MySQL root user: [root]rootEnter MySQL root password: []7layerDo you want to clone a database from Gateway2.l7tech.com (yes or no)? [yes]yesEnter name of database to clone: [ssg]ssg--> MASTER = Gateway2.l7tech.com--> DBUSER = repluser--> DBPWD = replpass--> ROOT = root--> ROOT_PWD = 7layer--> CLONE_DB = yes--> DB = ssg--> Stopping slave--> File = ssgbin-log.000002--> Position = 62095412--> Changing MASTER settings--> Creating database: ssg--> Copying database from Gateway2.l7tech.com--> Starting slave--> Confirming slave startup--> Slave_IO_Running = Yes--> Slave_SQL_Running = YesSlave successfully created--> Getting password for user gateway --> Granting access for 'gateway'@'localhost' --> Granting access for 'gateway'@'%' --> Granting access for 'gateway'@'localhost.localdomain' --> Granting monitor rights for monitor@localhostDoes the clone command time out? If so, see Troubleshooting.
- Exit the privileged shell to the Gateway main menu.Select option2(Display CA API Gateway configuration menu) and then option3(Configure the CA API Gateway).
- Configure the DR Gateway to use the localhost database but remain in a disabled state:
- ---------------------------------------------------------------------- Configure CA API Gateway ---------------------------------------------------------------------- At any time type "quit" to quit.----------------------- Set Up the CA API Gateway Database ----------------------- At any time type "quit" to quit. Press "<" to go to the previous step.This step lets you create or set up a connection to the SSG database.Enter the database hostname.Database Host [localhost]:localhostEnter the database port.Database Port [3306]:3306Enter the database name.Database Name [ssg]:ssgEnter the database user.Database Username [gateway]:gatewayEnter the database password.Database Password:db_passwordConfirm Database Password:db_password-------------------------------- Set Up the CA API Gateway Failover Database -------------------------------- At any time type "quit" to quit. Press "<" to go to the previous step.This step lets you create or set up a connection to the SSG failover database.This step is optional, enter "Yes" to continue or "No" to skip.Configure Database Failover Connection? [No]:No---------------------- Set Up the CA API Gateway Cluster ---------------------- At any time type "quit" to quit. Press "<" to go to the previous step.This step lets you set up the SSG cluster.Enter the cluster passphrase (6-128 characters).Cluster Passphrase:cluster_passphraseConfirm Cluster Passphrase:cluster_passphrase------------------- Set Up the SSG Node ------------------- At any time type "quit" to quit. Press "<" to go to the previous step.This step lets you set up the SSG node.Enable or disable the node.Enabled [Yes]:No--------------------- Configuration Summary --------------------- Press < to go to the previous step, type "quit" to quit.The following configuration will be applied:Database Connection Database Host = localhost Database Port = 3306 Database Name = ssg Database Username = gatewayNode Configuration Enabled = NoPress [Enter] to continue. Please wait while the configuration is applied ... --------------------- Configuration Results ---------------------The configuration was successfully applied.Press [Enter] to continue
Monitoring the Replication State
Required Script:
monitor_replication.sh
- Usescpto copy themonitor_replication.shscript to the home directory of thessgconfiguser.
- Open a privileged shell on Gateway-DR.
- Run the following commands to change the ownership and permissions of themonitor_replication.shscript to the home directory of the ssgconfig user:#chown root:root ~ssgconfig/monitor_replication.sh#chmod 700 ~ssgconfig/monitor_replication.sh#mv ~ssgconfig/monitor_replication.sh /usr/local/bin
- Open/usr/local/bin/monitor_replication.shfor editing and set the configurable settings, including:SettingNotesMONUSERMust be same as value increate_DR_slave.shMONPWDMust be same as value increate_DR_slave.shNOTIFYUse default unless you have a specific setting in mindNOTIFY_SMTPConfigureeximas per notes in script→ NOTIFY_TORequired→ NOTIFY_CCOptional→ NOTIFY_BCCOptional→ MTA_FLAGSee notes for value; use default unless you have a specific setting in mindNOTIFY_SNMPIf you use SNMP for notification, ensure thatnet-snmp-utilsis installed. For more information, see the notes within the script.→ SNMP_HOSTRequired→ COMMUNITYRequired→ OID_ERRORUse default unless you have a specific setting in mindVERBOSESet to 'yes' for testing and 'no' for production
- Test themonitor_replication.shscript file:#monitor_replication.shMaster_Host = Gateway2.l7tech.com Slave_IO_Running = Yes Slave_SQL_Running = Yes Master_Log_File = ssgbin-log.000005Slave is functioning properly.#mysqladmin stop-slaveSlave stopped #monitor_replication.shMaster_Host = Gateway2.l7tech.com Slave_IO_Running = No Slave_SQL_Running = No Master_Log_File = ssgbin-log.000005WARNING: Slave is not functioning properly --- This *should* have been sent by SMTP or SNMP --- Sending notification by SNMP trap #mysqladmin start-slaveSlave started #monitor_replication.shMaster_Host = Gateway2.l7tech.com Slave_IO_Running = Yes Slave_SQL_Running = Yes Master_Log_File = ssgbin-log.000005Slave is functioning properly.
- In a production environment, edit/usr/local/bin/monitor_replication.shand setVERBOSE='no'.
- Add/usr/local/bin/monitor_replication.shto crontab. For more information, see the notes within the script for more information.#crontab -e# Monitor DR database replication0 * * * * /usr/local/bin/monitor_replication.sh
Troubleshooting
Resolving Clone Timeouts
If the database is very large, the clone command may time out. If this happens, access a privileged shell on Gateway2 and manually dump the database with:
# mysqldump --master-data=1 -r ssgdump.sql ssg
Then copy
ssgdump.sql
to the Gateway-DR and load it:# mysqladmin stop-slave# mysqladmin create ssg# mysql ssg < ssgdump.sql# mysqladmin start-slave# mysql -e "SHOW SLAVE STATUS\G" | grep Running Slave_IO_Running: Yes Slave_SQL_Running: Yes
Next step:
Activating the Failover Node