История страницы

...

For correct operation, it is required to provide L2 connectivity between two remote servers.

Installing and configuring keepalived

Package description

Keepalived package is an open source software used to perform high availability and load balancing functions. The first function is based on the implementation of the VRRP protocol, and the second is based on the Linux Virtual Server (IPVS) kernel module. Keepalived is not developed by Eltex and does not include adjustments other than configuration. Keepalived is used to provide backup for SoftWLC controllers with only VRRP functions.

Installing keepalived

In order to install the package, download it on the server and run the following command (installation must be done under the root superuser on both servers):

...

! Configuration File for keepalived
 
global_defs {
 
   script_user root
   enable_script_security 
}

vrrp_script check_network {
    script "/etc/keepalived/check_ping.sh"
    interval 5
    weight 50
    fall 3
    rise 3
    init_fail
    user root
}

vrrp_instance VI_SWLC {
    state BACKUP
    interface <interface>
    virtual_router_id 1
    track_script {
        check_network
    }
    track_interface {
        <interface> weight 50
    }
    priority 150
    advert_int 1
    nopreempt
	# Uncomment and comment "nopreempt" if preemption needed
	#preempt_delay 180
    authentication {
        auth_type PASS
        auth_pass eltex
    }
    virtual_ipaddress {
        <virtual_ip> dev <interface> label <interface>:1
    }
 
    notify_master "/etc/keepalived/keep_notify.sh master"
    notify_backup "/etc/keepalived/keep_notify.sh backup"
    notify_fault "/etc/keepalived/keep_notify.sh fault"
 
    unicast_peer {
        <ip_address_other_servers>
    }
}

Test script

The script pings a default gateway and returns the result code. Thus, SoftWLC is guaranteed to be accessible for external clients if the script has been executed successfully. 

...

where <default_gw_ip> is the default gateway for this server, similar to the entry (100.10.194.1);.

Configuring role change

When the server state changes, the script keep_notify.sh is executed where <mysql_user> and <mysql_password> are the login and password from the MySQL database (by default root/root).

#!/bin/bash

MYSQL_USER="<mysql_user>"
MYSQL_PASSWORD="<mysql_password>"

mongo_set_role() {
    local role="$1"
    if [[ "$(which mongo)" ]]; then
        mongo --quiet --eval "var role=\"$role\"" admin /etc/keepalived/mongo_switch.js
        # Uncomment if using mongodb auth
        #mongo -u<username> -p<password> --quiet --eval "var role=\"$role\"" admin /etc/keepalived/mongo_switch.js
    fi
}

if ! lockfile-create --use-pid -r 5 /tmp/keep.mode.lock; then
    echo "Unable to lock"
    echo "Unable to lock" > /tmp/keep.mode.lock.fail
    exit 0
fi

case "$1" in
    master)
    #  ems_reload_all
    echo "MASTER" > /tmp/keep.mode
  
    mongo_set_role master
    service eltex-ems restart
    service tomcat8 restart
    service eltex-ngw restart

    # restart MySQL slave to get updates immediately after reestablishing communication,
    # rather than wait for a heartbeat from the second server
    mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -e "stop slave"
    mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -e "start slave"
  ;;
 backup)
    echo "BACKUP" > /tmp/keep.mode
    mongo_set_role slave
    service mongod restart
    service eltex-ems stop
    service tomcat8 stop
    service eltex-ngw stop
    mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -e "stop slave"
    mysql -u$MYSQL_USER -p$MYSQL_PASSWORD -e "start slave"
 ;;
 fault)
    echo "FAULT" > /tmp/keep.mode
    mongo_set_role slave
    service mongod restart
 ;;
 *)
    echo "Usage: $0 {master|backup|fault}"
    exit 1
esac

lockfile-remove /tmp/keep.mode.lock;

exit 0

replicaSet MongoDB master changing script

// provided by environment
var role;

if (role != 'master' && role != 'slave') {
    throw "Role must be either master or slave";
}

var thisIsMaster = (role == 'master');
var status = rs.isMaster();
var thisHost = status.me;

print("Primary: " + status.ismaster + "; applying configuration ...");
var cfg = rs.conf();
for (var i = 0; i < cfg.members.length; i++) {
    var member = cfg.members[i];
    var self = (member.host == thisHost);
    if (self ^ thisIsMaster) {
        // Configuration for slave
        member.priority = 1;
        member.votes = 0;

        print(member.host + ": secondary");
    } else {
        // Configuration for master
        member.priority = 2;
        member.votes = 1;

        print(member.host + ": primary");
    }
}

var result = rs.reconfig(cfg, { force: !status.ismaster });
if (result.ok == 1) {
    print("Reconfiguration done");
} else {
    print(result);
}

...

admin@swlc01-server:/# sudo chmod +x /etc/keepalived/check_ping.sh
admin@swlc01-server:/# sudo chmod +x /etc/keepalived/keep_notify.sh
admin@swlc01-server:/# sudo chmod +x /etc/keepalived/mongo_switch.js

Saving log to separate file

By default, keepalived saves a log to the /var/log/syslog file. For keepalived debugging, monitoring and managing convenience, separate log filing can be configured.

...

Now messages from the keepalived daemon will only get into the log file /var/log/keepalived.log and will not get into /var/log/syslog.

Starting/stopping keepalived procedure

To start the service, run the following command:

...

Configuring rsync

Rsync in the backup scheme is responsible for synchronizing service files, Eltex-EMS and Eltex-APB services, as well as firmware files, configuration templates, point configuration uploads. Rsync is a client-server software. Master server acts as a client and synchronizes slave server's directories with local ones.

Configuring rsync server

To enable the rsync server, it is necessary to set the value RSYNC_ENABLE=true on each server in the /etc/default/rsync file:

...

root@swlc01-server:/# crontab -e

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.tiny
  3. /usr/bin/code
  4. /bin/ed

Choose 1-4 [1]: 1                                 # choose an editor

Starting/stopping procedure

To start the service, use the command:

...

OK. Successful synchronization of files from directory: /usr/lib/eltex-ems/conf/* 
OK. Successful synchronization of files from directory: /tftpboot/* 
OK. Successful synchronization of files from directory: /var/ems-data/WP/*

Configuring MySQL replication

Backup of data stored in MySQL database is carried out by master-master replication. That means each server is both master and slave at the same time. The scheme implies writing all database updates of the first server to a special binary log. The second server reads the log and applies the changes.   The second server replicates data from the first, and the first from the second. That allows having a relevant copy of a database on two hosts simultaneously. If connection fails, changes are accumulated and then synchronized after reconnection.

Data dump transferring and transferring to the second server

When configuring backup during operation (i.e. if the current server's MySQL already has data in it), it is necessary to replicate data to the second server. This can be done using the mysqldump utility.

...

root@swlc01-server:/# mysql -uroot -proot < /home/<username>/mysqldump_master.sql

Configuring MySQL

Mysql daemon configuration aims at specifying binary logs writing parameters. The words "first server" and "second server" are further conditional and used to refer to differences in server configurations:

...

admin@swlc01-server:/# sudo service mysql restart

Creating user accounts

For replication to work, a service account should be created on both servers. The server will connect master server and get data changes using this account.

...

Granting rights to service users

Open /usr/lib/eltex-ems/conf/config.txt , see which username/password are used (by default - javauser / javapassword)

...

GRANT ALL PRIVILEGES ON *.* TO 'javauser'@'%' IDENTIFIED BY 'javapassword';
GRANT ALL PRIVILEGES ON eltex_auth_service.* TO 'javauser'@'%'; 
GRANT ALL PRIVILEGES ON `radius`.* TO 'javauser'@'%';             
GRANT ALL PRIVILEGES ON `wireless`.* TO 'javauser'@'%';           
GRANT ALL PRIVILEGES ON `Syslog`.* TO 'javauser'@'%';             
GRANT ALL PRIVILEGES ON `eltex_doors`.* TO 'javauser'@'%';        
GRANT ALL PRIVILEGES ON `eltex_ngw`.* TO 'javauser'@'%';          
GRANT ALL PRIVILEGES ON `ELTEX_PORTAL`.* TO 'javauser'@'%';       
GRANT ALL PRIVILEGES ON `eltex_ems`.* TO 'javauser'@'%';          
GRANT ALL PRIVILEGES ON `eltex_alert`.* TO 'javauser'@'%';        
GRANT ALL PRIVILEGES ON `eltex_auth_service`.* TO 'javauser'@'%';
FLUSH PRIVILEGES;

Starting replication

Starting replication on the second server

Run the show master status command in MySQL console on the first server and analyze the values obtained:

...

If the Slave_IO_Running and Slave_SQL_Running parameters are set to "Yes", replication has started successfully.

Starting replication on the first server

On the second server, run:

mysql> show master status \G

*************************** 1. row ***************************
            File: mysql-bin.000001
        Position: 00000107
    Binlog_Do_DB: eltex_alert,eltex_ems,eltex_ont,radius,wireless,eltex_auth_service,payments,ELTEX_PORTAL
Binlog_Ignore_DB: mysql,Syslog,performance_schema,information_schema
1 row in set (0.00 sec)

...

If the Slave_IO_Running and Slave_SQL_Running parameters are set to "Yes", the values of Master_Log_File and Read_Master_Log_Pos are replicated in both directions.

Checking replication from EMS-GUI

MySQL replication state can be controlled from GUI EMS. To do this, edit the configuration file /etc/eltex-ems/check-ems-replication.conf. Changes must be made on both servers.

...

Configuring MongoDB

In MongoDB, replication is performed by grouping several (3 for standard configuration) nodes into Replica Set. Replica Set consists of one primary node and several secondary nodes. The following scheme explains it in details:

...

Installing mongodb on arbiter

For replication to work correctly, MongoDB versions must match on all hosts. For standard mongo installation, version 3.6.3 is required, for the example below, version 4 is required. 

...

sudo systemctl enable mongod.service
sudo systemctl start mongod.service

replicaSet configuration

In /etc/mongod.conf on all nodes:

...

To check the Replica Set status, run the rs.status()command in the MongoDB console.

Adding/deleting/changing nodes in Replica Set

Configuration of nodes in Replica Set can be performed only on PRIMARY.

...

Eltex-PCRF operation in cluster mode

Configuring PCRF cluster

Open 5701 tcp and 5801 tcp ports between PCRF servers. 

...

When using a PCRF cluster on ESR, configure interaction with all nodes of the cluster using their real address.

Configuring SoftWLC modules

It is necessary to configure SoftWLC modules on both servers to interact with controller via virtual ip. The following configuration files should be modified.

...

• Change localhost to <virtualip> in the lines 4, 17, 26, 35, 48, 57, 66, 75, 84, 98.

Adding user to NAS table

To access your Admin Panel, you need to add the appropriate entries to the NAS table.

...

SoftWLC modules should also be configured via graphical interface.

Admin Panel

In the section Settings → Integration in the parameters PCRF url, NGW Client url and Portal constructor URL, change localhost to a virtual ip address:

Portal Constructor

Replace localhost with a virtual IP address in the following sections:

...

System settings → Mercury access

EMS-GUI

In EMS GUI, replace localhost (or 127.0.0.1) with a virtual IP address in the following sections:

...