General information
SoftWLC controller backup is performed according to the master-slave model. Files critical for system operation (configuration files, firmware files, upload data files), MySQL database files (in master-master mode), MongoDB database files and DHCP server logs are synchronized. Such a model provides service availability and relevance of data on both controllers if one of them fails, network is inaccessible or power supply problems occur.
In configuration examples of this section, IP addresses will be referred to as <ip_server1>, <ip_server2> and <virtual_ip>, where:
- <ip_server1> — real ip address of the first server
- <ip_server2> — real ip address of the second server
- <virtual_ip> — virtual ip address
SoftWLC controllers backup configuration includes the following steps:
- installing and configuring keepalived
- configuring rsync
- Configuring MySQL replication
- configuring replicaSet MongoDB
- configuring Eltex-PCRF operation in cluster mode
- changing configuration of modules for them to use virtual IP
Installing and configuring keepalived
The main component for controller backup. Provides master role passing and system operation problems detecting. To install keepalived, contact the Wi-Fi Service center and get a relevant distribution kit.
Configuring rsync
In backup scheme, rsync manages synchronization of Eltex-EMS and Eltex-APB service files, and also firmware, configuration and AP configuration upload files. Rsync is a client-server software. Master server acts as a client and synchronizes slave server's directories with local ones.
Starting/stopping procedure
To enable rsync server, define the following value in the file /etc/default/rsync:
RSYNC_ENABLE=true
To start the service after stopping:
root@swlc01-server:/# service rsync start
To stop the service, the following command is used:
root@swlc01-server:/# service rsync stop
To check the service status, use the command:
root@swlc01-server:/# service rsync status
This will be followed by a message:
* rsync is running
if the service is running, or by
* rsync is not running
if it is not.
Configuring rsync server
The main configuration file of rsync server is located in /etc/rsyncd.conf. The listing is given below.
[ems-conf] path = /usr/lib/eltex-ems/conf/ use chroot = no max connections = 2 lock file = /var/lock/rsyncd read only = no list = no uid = root auth users = backup secrets file = /etc/rsyncd.secrets strict modes = yes # IP-address of the server that will access the resource, i.e., of the second server in the pair hosts allow = <ip_server1> <virtual_ip> ignore errors = no ignore nonreadable = yes transfer logging = no timeout = 60 refuse options = checksum dry-run dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz [ems-tftp] path = /tftpboot use chroot = no max connections = 2 lock file = /var/lock/rsyncd.tftp read only = no list = no uid = root auth users = backup secrets file = /etc/rsyncd.secrets strict modes = yes hosts allow = <ip_server1> <virtual_ip> ignore errors = no ignore nonreadable = yes transfer logging = no timeout = 60 refuse options = checksum dry-run dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz [ems-wp] path = /var/ems-data/WP use chroot = no max connections = 2 lock file = /var/lock/rsyncd.ems-wp read only = no list = no uid = root auth users = backup secrets file = /etc/rsyncd.secrets strict modes = yes hosts allow = 10.62.8.121 10.62.8.122 ignore errors = no ignore nonreadable = yes transfer logging = no timeout = 60 refuse options = checksum dry-run dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz
hosts allow parameters are specified for master server. The following form is recommended:
hosts allow = <other_server_ip> <virtual ip>
To perform authentication, configure *rsync* user on both servers by creating files /etc/rsyncd.secrets on both servers and specifying login and password in them.
backup:rspasswd
Assign file access rights by running the following commands on both servers:
root@swlc01-server:/# chmod 600 /etc/rsyncd.secrets
Configuring synchronization launch
Create files /etc/rsync_client.secrets, specify the following password there:
root@swlc01-server:/# echo "rspasswd" > /etc/rsync_client.secrets && chmod 600 /etc/rsync_client.secrets
File synchronization is performed by cron task where /usr/lib/eltex-ems/scripts/rsync_ems_backup.sh is run. The script starts rsync client and synchronizes local directories with directories of the second (backup) server.
Synchronization can be started only if the server works in master state.
#!/bin/bash
LOCKFILE="/run/lock/rsync_ems_backup"
# IP address backup server
HOST=<ip_server2>
# Check if we're root
if [ `whoami` != "root" ]
then
echo "This script should be run by root."
exit 1
fi
# Check and create lock file
if ! lockfile-create --use-pid -r 0 $LOCKFILE &> /dev/null ; then
echo "Backup is already running"
exit 0
fi
# Check - if we're master - try to perform backup to slave
SRVMODE=`cat /tmp/keep.mode`
if [ "$SRVMODE" == "MASTER" ]
then
rsync -urlogtp --delete-after --password-file=/etc/rsync_client.secrets /usr/lib/eltex-ems/conf/ backup@$HOST::ems-conf > /tmp/rsync_ems_conf.log 2>&1
echo $? >> /tmp/rsync_ems_conf_result.log
rsync -urlogtp --delete-after --password-file=/etc/rsync_client.secrets /tftpboot/ backup@$HOST::ems-tftp > /tmp/rsync_ems_tftpboot.log 2>&1
echo $? >> /tmp/rsync_ems_tftpboot_result.log
rsync -urlogtp --delete-after --password-file=/etc/rsync_client.secrets /var/ems-data/WP/ backup@$HOST::ems-wp > /tmp/rsync_ems_wp.log 2>&1
echo $? >> /tmp/rsync_ems_wp_result.log
else
echo "Not master. No action will be performed."
fi
lockfile-remove $LOCKFILE
where
backup– login specified in /etc/rsyncd.secrets fileHOST– another server's IP address
Create cron tasks on both servers to start synchronization every minute:
root@swlc01-server:/# crontab -l | { cat; echo "*/1 * * * * /usr/lib/eltex-ems/scripts/rsync_ems_backup.sh"; } | crontab
cron service should be started on both servers
Configuring MySQL replication
Backup of data stored in MySQL database is carried out by master-master replication. That means each server is both master and slave at the same time. The scheme implies writing all database updates of the first server to a special binary log. The second server reads the log and applies the changes. The second server replicates data from the first server, and vice versa (http://dev.mysql.com/doc/refman/5.5/en/replication.html). That allows having a relevant copy of a database on two hosts simultaneously. If connection fails, changes are accumulated and then synchronized after reconnection.
Data dump transfer and transfer to the second server
When configuring backup during operation (i.e. if the current server's MySQL already has data in it), it is necessary to replicate data to the second server. This can be done using the mysqldump utility.
Block the tables, take the dump, unblock the tables and copy the file to the second server:
root@swlc01-server:/# mysql -uroot -proot -e "FLUSH TABLES WITH READ LOCK;" root@swlc01-server:/# mysqldump -uroot -proot --databases ELTEX_PORTAL eltex_alert eltex_auth_service eltex_ems payments radius wireless > mysqldump_master.sql root@swlc01-server:/# mysql -uroot -proot -e "UNLOCK TABLES;" root@swlc01-server:/# scp mysqldump_master.sql <username>@<ip_server2>:/home/<username>/
Then generate a dump on the second server:
root@swlc01-server:/# mysql -uroot -proot < /home/<username>/mysqldump_master.sql
MySQL configuration
mysql daemon configuration aims at specifying binary logs writing parameters. The words first server and second server are conditional and used to refer to differences in server configurations.
Make the following changes in [mysqld] section of /etc/mysql/my.cnf configuration file:
Comment out or delete the following line on both servers:
bind-address = 127.0.0.1
Specify server-id. The servers should be given unique identificators, e.g., for the first server:
server-id = 1
For the second server:
server-id = 2
Enable binary logs on both servers:
log_bin = /var/log/mysql/mysql-bin.log
specify auto_increment_increment (increment step) and auto_increment_offset (start point) parameters.
For the first server:
auto_increment_increment= 2 auto_increment_offset = 1
For the second server:
auto_increment_increment= 2 auto_increment_offset = 2
For both servers:
- specify databases for which logs will be written:
binlog-do-db = eltex_alert binlog-do-db = eltex_ems binlog-do-db = wireless binlog-do-db = radius binlog-do-db = eltex_auth_service binlog-do-db = ELTEX_PORTAL binlog-do-db = payments
- specify databases for which logs will not be written:
binlog-ignore-db = mysql binlog-ignore-db = Syslog binlog-ignore-db = performance_schema binlog-ignore-db = information_schema
Restart mysql on each server and create a database for replication.
root@swlc01-server:/# service mysql restart
Creating user accounts
For replication to work, a service account should be created on both servers. The server will connect master server and get data changes using this account.
Create an account for replication on the first server:
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replication'@'<ip_server2>' IDENTIFIED BY 'password'; FLUSH PRIVILEGES;
Create an account for replication on the second server:
GRANT SELECT, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'replication'@'<ip_server1>' IDENTIFIED BY 'password'; FLUSH PRIVILEGES;
SELECT privilege is used to check replication performance from GUI EMS
Starting replication
Starting replication on the second server
Run the show master status command in MySQL console of the first server and analyze the values obtained:
mysql> show master status \G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 00000107
Binlog_Do_DB: eltex_alert,eltex_ems,radius,wireless,eltex_auth_service,payments,ELTEX_PORTAL
Binlog_Ignore_DB: mysql,Syslog,performance_schema,information_schema
1 row in set (0.00 sec)
Remember the parameters File and Position.
It is recommended to set Position equal to 107. This is the position from which log file writing begins.
Configure and start the second server replication from the first one (perform the following operations on the second server):
mysql> STOP SLAVE; mysql> CHANGE MASTER TO MASTER_HOST='<ip_server1>', MASTER_USER='replication', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=107; mysql> START SLAVE;
where
- MASTER_LOG_FILE='mysql-bin.000001' – specify File value, got on the first server.
- MASTER_LOG_POS=107 – specify Position value, got on the previous step (on the initial setting, 107 is recommended).
Check replication state on the second server:
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: <ip_server1>
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 107
Relay_Log_File: mysqld-relay-bin.000001
Relay_Log_Pos: 107
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 0
Last_Error:
Skip_Counter: 0
Exec_Master_Log_Pos: 107
Relay_Log_Space: 107
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 0
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id: 2
1 row in set (0.00 sec)
If Slave_IO_Running and Slave_SQL_Running are set to «Yes», replication has been started successfully.
Starting replication on the first server
On the second server run:
show master status \G
mysql> show master status \G
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 00000107
Binlog_Do_DB: eltex_alert,eltex_ems,eltex_ont,radius,wireless,eltex_auth_service,payments,ELTEX_PORTAL
Binlog_Ignore_DB: mysql,Syslog,performance_schema,information_schema
1 row in set (0.00 sec)
Configure and start first server replication from the second server (run the following commands on the first server):
CHANGE MASTER TO MASTER_HOST='<ip_server2>', MASTER_USER='replication', MASTER_PASSWORD='password', MASTER_LOG_FILE='mysql-bin.000001', MASTER_LOG_POS=107; START SLAVE;
Check replication state on the first server:
mysql> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: <ip_server2>
Master_User: replication
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: mysql-bin.000001
Read_Master_Log_Pos: 107
Relay_Log_File: mysqld-relay-bin.000001
Relay_Log_Pos: 107
Relay_Master_Log_File: mysql-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
...
the output given is incomplete, since other data is less important
If Slave_IO_Running and Slave_SQL_Running parameters are set to «Yes», Master_Log_File and Read_Master_Log_Pos values are given, replication is performed in both directions.
Checking replication from EMS-GUI
MySQL replication state can be controlled from GUI EMS. To do that, edit /etc/eltex-ems/check-ems-replication.conf configuration file. The changes should be made on both servers:
# Enable("Yes") / Disable("No") replication check
ENABLE_REPLICATION="Yes"
# The first replication host's address
HOST1=<ip_server1>
# The second replication host's address
HOST2=<ip_server2>
# mysql server access parameters
# mysql user
USER="replication"
# mysql password
PASSWORD="password"
where
- ENABLE_REPLICATION shows whether replication check is enabled (set to "Yes")
HOST1, HOST2- servers' IP addressesUSER, PASSWORD- user account login/password for replication.
After saving the changes, replication state can be checked via GUI EMS in Information → State of backup system → MySQL.
Results of replication state check on both servers and brief summary of check results are given in the section.
Configuring MongoDB
In MongoDB, replication is performed via grouping several (3 for standart configuration) nodes into Replica Set. Replica Set consists of one Primary node and several Secondary nodes (more information on https://docs.mongodb.com/v2.4/administration/replica-sets.
All data change operations are performed only on Primary. Thus, MongoDB automatically performs failover and replaces Primary with a working node if current Primary fails. But that requires 3+ nodes in Replica Set.
In the default configuration, Replica Set that consists of two nodes completely goes down when a failure in one of them (even in Secondary).
replicaSet configuration
In /etc/mongodb.conf on both nodes:
add the line
replSet = <replica_set_name>
where <replica_set_name> is a name of replica set, which is chosen arbitrarily but should be the same for both servers.
Comment out or delete the line:
# bind_ip = 127.0.0.1
Restart MongoDB
root@swlc01-server:/# service mongodb restart
Open MongoDB console on the first node
root@swlc01-server:/# mongo
Create replica set configuration
> rs.initiate()
After a while, shell prompt should be changed to:
replica_set_name:PRIMARY>
If DNS is not used in the network, check if the first node was successfully added to Replica Set configuration.
replica_set_name:PRIMARY> rs.config()
{
"_id" : "replica_set_name",
"version" : 63243,
"members" : [
{
"_id" : 0,
"host" : "<hostname_server1>:27017"
}
]
}
If the first element (at index 0) has hostname instead of IP address, and DNS is not used in the network, Replica Set configuration should be updated before adding the second node. If DNS is used, and DNS name is correct, the second node can be added.
To update the server's address, run the following commands:
replica_set_name:PRIMARY> conf = rs.conf() replica_set_name:PRIMARY> conf.members[<индекс>].host = "<ip_server1>:27017" replica_set_name:PRIMARY> rs.reconfig(conf)
Check the current configuration again:
replica_set_name:PRIMARY> rs.config()
{
"_id" : "relica_set_name",
"version" : 63243,
"members" : [
{
"_id" : 0,
"host" : "<ip_server1>:27017"
}
]
}
"host" parameter should contain the server's IP address.
Add the second node to Replica Set (run on the first server):
replica_set_name:PRIMARY> rs.add("<ip_server2>")
{ "ok" : 1 }
MongoDB error response can be generated if there is no connection to the second node (or bind_ip = 127.0.0.1 is specified there), or replSet is not configured .On the second node, MongoDB management console prompt should be changed to:
root@swlc01-server:/# mongo replica_set_name:SECONDARY>
The same should be done for other nodes. Replica Set state can be checked by running the command rs.status() in MongoDB console.
Eltex-PCRF operation in cluster mode
Configuring PCRF cluster
Open 5701 tcp and 5801 tcp ports between PCRF servers
On servers, in /etc/eltex-pcrf/hazelcast-cluster-network.xml configuration files, specify network interfaces' addresses (lines 5 and 22 of the example contain the server's address, and lines 14 and 15 contain the list of all cluster members).
Configuration example:
<network>
<!-- Write here public address of the node -->
<!-- specify the server's own address here -->
<public-address>ip_server1</public-address>
<port auto-increment="false" port-count="100">5701</port>
<outbound-ports>
<ports>0</ports>
</outbound-ports>
<join>
<multicast enabled="false"/>
<tcp-ip enabled="true">
<!-- List IP addresses of all cluster members (including this one) -->
<member>ip_server1</member>
<member>ip_server2</member>
</tcp-ip>
<discovery-strategies>
</discovery-strategies>
</join>
<interfaces enabled="true">
<!-- specify the server's own address here -->
<interface>ip_server1</interface>
</interfaces>
Permit cluster start in /etc/eltex-pcrf/eltex-pcrf.json configuration file:
"cluster.enable" : true,
Restart Eltex-PCRF using the command
root@swlc01-server:/# service eltex-pcrf restart
Cluster state check
{
"data" : {
"enabled" : true,
"state" : "ACTIVE",
"members" : [ {
"address" : "ip_server1",
"local" : true,
"active" : true
}, {
"address" : "ip_server2",
"local" : false,
"active" : true
} ],
"messagesStats" : {
"received" : 45157,
"sent" : 45144
},
"mongo" : {
"available" : false,
"error" : "not running with --replSet"
}
},
"key" : "PcrfErrorCode.success",
"message" : "Success",
"code" : 0,
"args" : [ ]
}
Configuring SoftWLC modules
It is necessary to configure SoftWLC modules on both servers to interact with controller via virtual ip. The following configuration files should be changed:
- Change
localhostto<virtual_ip>in line 2.
- Change
localhostto<virtualip>in lines 28, 35, 61, 68.
Change localhost to <virtualip> in the lines 9, 36.
Change localhost to <virtualip> in line 18.
- Change
localhostto<virtualip>in lines 3, 11, 19, 27, 35, 71, 77, 81, 85, 101 - Change
127.0.0.1to<virtualip>in lines 49, 66
- Change
localhostto<virtualip>in line 19
- Change
localhostto<virtualip>in lines 7, 18 - Change
127.0.0.1to<virtualip>in line 25
- Change
localhostto<virtualip>in lines 4, 5, 9
Change localhost to <virtualip> in the lines 4, 17, 26, 35, 48, 57, 66, 75, 84, 98
Changing configuration via GUI
SoftWLC modules should also be configured via graphical interface.
Admin Panel
In the section Settings → Integration replace localhost with a virtual IP address in PCRF URL and URL NGW-клиента parameters:
Portal Constructor
Replace localhost with a virtual IP address in the following sections:
System settings → Portal Constructor
System settings → NBI access
System settings → NGW access
System settings → Payments DB
System settings → PCRF access
EMS-GUI
In EMS GUI, replace localhost (or 127.0.0.1) with a virtual IP address in the following sections:
Administration → EMS server configuration → System modules settings → pcrf
Administration → EMS server configuration → System modules settings → radius
Administration → EMS server configuration → System modules settings → softwlc.nbi
Administration → EMS server configuration → System modules settings → system
Administration → EMS server configuration → System modules settings → tftpserver
