Update from version 3.11 to 3.14
It is recommended to update from a new installation of the operating system (Ubuntu Server 18.04.x LTS 64bit) and new disk space layout.
This section describes the process of updating the operating system and ECSS-10 software components, when for some reason a new installation is not possible.
Preparing to update the system
Update the system to the latest versions:
sudo apt update sudo apt upgrade
Perform a full backup (saving directories to /etc, /var/lib/ecss, /var/lib/ecss-mysql, /usr/share/ecss).
If you need the latest billing data, then finalize the current cdr and download it to a reliable place, for example:
admin@mycelium1@ecss1:/$ domain/bsk.test/cdr/make_finalize_cdr bsk3 Created file(s): cdrbsk3_20210330_11_05_38_t.csv
View how many registered subscribers, example:
admin@mycelium1@ecss1:/$ domain/bsk.test/sip/user/registered * .... ┌───────────────┐ │ elements: 320 │ └───────────────┘
When updating the cluster, the servers should be updated in turn. Thus, one of the servers will always be running and the services will not be interrupted during update. To separate the cluster, use ecss-control tool on both servers.
Example:
sasha@ecss1:~$ sudo ecss-control detach detach host: ecss2, ip: 10.10.0.2 complete sasha@ecss1:~$ sudo ecss-control status locked ip: 10.10.0.2
After splitting the cluster, it is necessary to turn off all nodes on the server that will be updated first:
sudo systemctl stop ecss-mycelium.service sudo systemctl stop ecss-ds.service sudo systemctl stop ecss-core.service sudo systemctl stop ecss-pa-sip.service sudo systemctl stop ecss-pa-megaco.service sudo systemctl stop ecss-mediator.service sudo systemctl stop ecss-web-conf.service sudo systemctl stop ecss-media-server.service sudo systemctl stop ecss-subscriber-portal.service sudo systemctl stop ecss-restfs.service sudo systemctl stop ecss-cc-ui-api.service
Remove ecss-3.11 packages, saving configurations and databases:
sudo apt remove ecss-*-3.11 sudo apt autoremove sudo apt upgrade
OS upgrade
Version 3.14 is installed on Ubuntu 18.04, and 3.11 is installed on Ubuntu 16.04. It means that OS on the server should be updated before installing a new version of ECSS-10. As soon as old ECSS-10 packages are removed, you can start upgrading the server:
If Linux kernel was upgraded during the update, the server should be restarted.
Once again make sure that all packages are updated and enter the OS upgrade command:
sudo apt upgrade sudo do-release-upgrade
If upgrade is done remotely via SSH, the system will ask if you want to continue and open another port for ssh — 1022. Thus, it will be possible to connect via this port in case of any problems.
During the upgrade, various questions will be asked, for example, whether you need to save an existing configuration file or install a version of the accompanying package. If there were no changes to the file, it should be entered safely (Y), otherwise the current configuration (N) will be saved.
Next, follow the on-screen instructions. During the update procedure, you may be prompted to replace the existing grub configuration file. Make sure that the current version has been saved to avoid possible problems.
After the upgrade is complete, the server needs to be restarted.
ECSS-10 version 3.14 installation
For more information, see ECSS packages installation.
Add version 3.14 repository to the system:
sudo sh -c "echo 'deb [arch=amd64] http://archive.eltex.org/ssw/bionic/3.14 stable main extras external' > /etc/apt/sources.list.d/eltex-ecss10-stable.list" sudo apt update
ecss-mysql installation
First, install ecss-mysql package, during which the databases will be migrated to the new version:
sudo sh -c "echo 'deb [arch=amd64] http://archive.eltex.org/ssw/bionic/3.14 stable main extras external' > /etc/apt/sources.list.d/eltex-ecss10-stable.list" sudo apt update
During the installation of ecss-mysql, you will be prompted to update the aparmor configuration. You should answer Yes (Y).
If there is a problem with any database, restart the aparmor service, then remove the database from mysql, and then install ecss-mysql again:
sudo systemctl restart apparmor.service mysql -uroot -p<root_mysql_password> -e 'DROP DATABASE <problematic database>' sudo apt install ecss-mysql
Configuring ecss-dns-env package
Before installing ecss-mysql package, you will be prompted to configure environment variables for services in dnsmasq. The customizer will prompt you to select the sections to configure. If the system is in a cluster, choose broker and mysql. The address of secondary broker is IP ecss2, mysql address is a "floating" address registered in keepalived.
If the system is not in a cluster, there is no need to select anything.
ecss-node installation
Install the ecss-node package:
sudo apt install ecss-node
SSL certificates are generated during package installation. You can leave the answers to the questions that are offered, including empty ones. For more information, see Operating system installation.
Installing the remaining packages
Install packages containing the main subsystems:
sudo apt install ecss-restfs ecss-media-server ecss-web-conf ecss-media-resources
When installing ecss-media-server, it is recommended to answer Yes to the configuration question. In version 3.14, new functionality has been added to MSR, so it is recommended to update the configuration file. If necessary, you can edit it manually later.
During the installation process, questions will be asked about saving the current versions of configuration files. It is recommended to install new ones, i.e. answer "Y".
The repository also contains additional packages that can be installed optionally based on the project.
To install additional packages, follow these steps:
sudo apt install <package name 1> <package name 1><package name 2> ... <package name N>
List of available additional packages:
ecss-cc-ui | Automated workplace of the Call center operator |
ecss-teleconference-ui | Automated workplace of the conference call manager |
ecss-utils | Scripts for converting binary logs to text |
ecss-asr | Automatic speech recognition service |
ecss-pda-api | API for Phone Desktop Assistant |
ecss-autoprovision | Automatic Telephone Configuration Service (AUP) |
ecss-clerk | Auto Secretary service |
ecss-crm-server | CRM Integration Server |
ecss-security | Service for logging user actions |
Here it will also be offered to replace all configuration files — it is necessary to agree (Y).
It is necessary to write the real name of the cluster after updating the configuration file /etc/ecss/ecss-mycelium/mycelium1.config, for example mycelium1:
sudo perl -pi -e 's/cluster_name, undefined/cluster_name, mycelium1/' /etc/ecss/ecss-mycelium/mycelium1.config cat /etc/ecss/ecss-mycelium/mycelium1.config | grep -e cluster_name
Check the services status, example:
admin@mycelium1@ecss1:/$ system-status Checking... ┌─┬───────────────┬─────────────────────────┬───────────────┬────────────┬──────┐ │ │ Node │ Release │ Erlang nodes │Mnesia nodes│Uptime│ ├─┼───────────────┼─────────────────────────┼───────────────┼────────────┼──────┤ │ │core1@ecss1 │ecss-core-3.14.10.220 │core1@ecss1 │not running │51m 6s│ │ │ds1@ecss1 │ecss-ds-3.14.10.220 │ds1@ecss1 │ds1@ecss1 │51m 6s│ │ │md1@ecss1 │ecss-mediator-3.14.10.220│md1@ecss1 │md1@ecss1 │51m 6s│ │ │mycelium1@ecss1│ecss-mycelium-14.10.220 │mycelium1@ecss1│not running │51m 6s│ │ │sip1@ecss1 │ecss-pa-sip-14.10.220 │sip1@ecss1 │sip1@ecss1 │51m 6s│ └─┴───────────────┴─────────────────────────┴───────────────┴────────────┴──────┘ All services are started. Active media resource selected list specific: ┌─────────────┬───────┬────────────┬───────────┬───────────┐ │ Node │ MSR │ MSR │ Cc-status │ Cc-uptime │ │ │ │ version │ │ │ ├─────────────┼───────┼────────────┼───────────┼───────────┤ │ core1@ecss1 │ msr_1 │ 3.14.10.67 │ connected │ 00:50:28 │ └─────────────┴───────┴────────────┴───────────┴───────────┘ Alarms list on md1: ┌────────────────┬──────────────────┬──────────────────┬────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────────────────────────────────────────────────────┐ │ Date │ Severity │ Location │ Cause │ Class │ Instance │ Message │ ├────────────────┼──────────────────┼──────────────────┼────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────────────────────────────────────────────────────┤ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │acb │Supplementary service acb has invalid version. You should update it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │sca │Supplementary service sca has invalid version. You should update it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │rbp │Supplementary service rbp has invalid version. You should update it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │cliro │Supplementary service cliro has invalid version. You should update it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │intervention │Supplementary service intervention has invalid version. You should updat│ │ │ │ │ │ │ │e it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │speed_dial │Supplementary service speed_dial has invalid version. You should update │ │ │ │ │ │ │ │it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │fax_to_email │Supplementary service fax_to_email has invalid version. You should updat│ │ │ │ │ │ │ │e it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │park │Supplementary service park has invalid version. You should update it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │3way │Supplementary service '3way' has invalid version. You should update it. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │introduce │Supplementary service introduce has invalid version. You should update i│ │ │ │ │ │ │ │t. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │sco_white │Supplementary service sco_white has invalid version. You should update i│ │ │ │ │ │ │ │t. │ │29.03 21:33:36 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │cc_agent │Supplementary service cc_agent has invalid version. You should update it│ │ │ │ │ │ │ │. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │second_handset │Supplementary service second_handset has invalid version. You should upd│ │ │ │ │ │ │ │ate it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │rfc │Supplementary service rfc has invalid version. You should update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │fbc │Supplementary service fbc has invalid version. You should update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │qsig_cfnr │Supplementary service qsig_cfnr has invalid version. You should update i│ │ │ │ │ │ │ │t. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │qsig_cfb │Supplementary service qsig_cfb has invalid version. You should update it│ │ │ │ │ │ │ │. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │fax_receiver │Supplementary service fax_receiver has invalid version. You should updat│ │ │ │ │ │ │ │e it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │teleconference_manager │Supplementary service teleconference_manager has invalid version. You sh│ │ │ │ │ │ │ │ould update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │clir │Supplementary service clir has invalid version. You should update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │pickup │Supplementary service pickup has invalid version. You should update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │cnip │Supplementary service cnip has invalid version. You should update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │clip │Supplementary service clip has invalid version. You should update it. │ │29.03 21:33:37 │warning │core1@ecss1 │versionMismatch │ecss::cluster::core::ss │cf_sip │Supplementary service cf_sip has invalid version. You should update it. │ │29.03 21:33:38 │critical │core1@ecss1 │outOfService │ecss::restfs::tts::statu│default_tts │The status of tts at restfs cluster "default" (http://system.restfs.ecss│ │ │ │ │ │s │ │:9990) is not available by reason 400 (Bad Request). │ └────────────────┴──────────────────┴──────────────────┴────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────────────────────────────────────────────────────┘
As seen, the versions of the services remained old. They need to be updated with the CLI command:
cluster/storage/ds1/ss/update *
Updating the second server if the system is in a cluster
To transfer traffic to the updated server, stop keepalived service on the second server:
sudo systemctl stop keepalived.service
For latest billing data, finalize the current cdr and download it to a reliable place, for example:
admin@mycelium1@ecss1:/$ domain/bsk.test/cdr/make_finalize_cdr bsk3 Created file(s): cdrbsk3_20220128_10_06_40_t.csv
Further update is similar to the description given above, taking into account the MSR settings for this host.
Building a cluster
After installing ecss-3.14 packages, ecss services should be stopped at this stage:
sudo systemctl stop ecss-mycelium.service sudo systemctl stop ecss-ds.service sudo systemctl stop ecss-core.service sudo systemctl stop ecss-pa-sip.service sudo systemctl stop ecss-pa-megaco.service sudo systemctl stop ecss-mediator.service sudo systemctl stop ecss-web-conf.service sudo systemctl stop ecss-media-server.service sudo systemctl stop ecss-restfs.service sudo systemctl stop ecss-cc-ui-api.service
Configuring dnsmasq
With a standard update, default configurations should remain — 127.0.0.1 is specified as IP address. The only thing that can be configured immediately when installing ecss-mysql package is the secondary.broker address. While the cluster is divided, the servers do not see each other.
When updating the system in cluster, IP addresses of both hosts must be specified in dnsmasq settings (in the example, 10.10.0.1 and 10.10.0.2 are static IP addresses of the hosts, and 10.10.0.15 VRRP is mysql IP address configured in keepalived). Then it is usually enough to edit the /etc/dnsmasq.d/ecss-mysql file on both servers.
Example of settings on ecss1/ecss2
sasha@ecss1:~/ecss1_3.14$ cat /etc/dnsmasq.d/ecss-broker address=/primary.broker.ecss/10.10.0.1 address=/secondary.broker.ecss/10.10.0.2 sasha@ecss1:~/ecss1_3.14$ cat /etc/dnsmasq.d/ecss-mysql address=/address-book.mysql.ecss/10.10.0.15 address=/cocon.mysql.ecss/10.10.0.15 address=/sp.mysql.ecss/10.10.0.15 address=/statistics.mysql.ecss/10.10.0.15 address=/tc.mysql.ecss/10.10.0.15 address=/tts.mysql.ecss/10.10.0.15 address=/dialer.mysql.ecss/10.10.0.15 address=/numbers.mysql.ecss/10.10.0.15
Then unnecessary files left from the previous version can be removed (they differ by the date of the last change). The following files should be kept:
sasha@ecss1:~/ecss1_3.14$ ls /etc/dnsmasq.d/ README ecss ecss-broker ecss-ldap ecss-mysql ecss-notifier ecss-restfs ecss-sip ecss-sp ecss-syslog lxd
To apply the settings, restart dnsmasq on both hosts and immediately check the status.
Example:
sasha@ecss1:~$ sudo systemctl restart dnsmasq.service
sasha@ecss1:~$
sasha@ecss1:~$ sudo systemctl status dnsmasq.service
● dnsmasq.service - dnsmasq - A lightweight DHCP and caching DNS server
Loaded: loaded (/lib/systemd/system/dnsmasq.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-03-30 15:06:53 +07; 31s ago
Process: 19252 ExecStop=/etc/init.d/dnsmasq systemd-stop-resolvconf (code=exited, status=0/SUCCESS)
Process: 19344 ExecStartPost=/etc/init.d/dnsmasq systemd-start-resolvconf (code=exited, status=0/SUCCESS)
Process: 19325 ExecStart=/etc/init.d/dnsmasq systemd-exec (code=exited, status=0/SUCCESS)
Process: 19304 ExecStartPre=/usr/sbin/dnsmasq --test (code=exited, status=0/SUCCESS)
Main PID: 19341 (dnsmasq)
Tasks: 1 (limit: 4616)
CGroup: /system.slice/dnsmasq.service
└─19341 /usr/sbin/dnsmasq -x /run/dnsmasq/dnsmasq.pid -u dnsmasq -r /run/dnsmasq/resolv.conf -7 /etc/dnsmasq.d,.dpkg-dist,.dpkg-old,.dpkg-new --local-service --trust-anchor=.,19036,8,2,49aac11d7b
мар 30 15:06:53 ecss1 systemd[1]: Starting dnsmasq - A lightweight DHCP and caching DNS server...
мар 30 15:06:53 ecss1 dnsmasq[19304]: dnsmasq: syntax check OK.
мар 30 15:06:53 ecss1 systemd[1]: Started dnsmasq - A lightweight DHCP and caching DNS server.
Starting the cluster
Run ecss services on the second host:
sudo systemctl start ecss-mycelium.service sudo systemctl start ecss-ds.service sudo systemctl start ecss-core.service sudo systemctl start ecss-pa-sip.service sudo systemctl start ecss-pa-megaco.service sudo systemctl start ecss-mediator.service sudo systemctl start ecss-web-conf.service sudo systemctl start ecss-media-server.service sudo systemctl start ecss-subscriber-portal.service sudo systemctl start ecss-restfs.service sudo systemctl start ecss-cc-ui-api.service
After starting all the services (check using systemctl status <service> command), the keepalived service can be started:
sudo systemctl start keepalived.service
Next, start MySQL database replication. Execute it on the first server. For more information, see MySQL master-master replication deployment scheme using keepalive.
Now you can open servers to each other to bring them into cluster. On the first server, run sudo ecss-control attach command.
Example:
sasha@ecss1:~/ecss1_3.14$ sudo ecss-control attach unlock ip: 10.10.0.2, rules: 7 complete sasha@ecss1:~/ecss1_3.14$ sudo ecss-control status all nodes attached
Checking operation of the system in the cluster
Check the update progress. system-status CLI command should show that all nodes in the cluster and subsystem versions are the same.
admin@mycelium1@ecss1$ system-status Checking... ┌─┬───────────────┬─────────────────────────┬───────────────────────────────┬─────────────────────┬──────┐ │ │ Node │ Release │ Erlang nodes │ Mnesia nodes │Uptime│ ├─┼───────────────┼─────────────────────────┼───────────────────────────────┼─────────────────────┼──────┤ │ │core1@ecss1 │ecss-core-3.14.10.220 │core1@ecss1,core1@ecss2 │not running │1h 19m│ │ │core1@ecss2 │ecss-core-3.14.10.220 │core1@ecss1,core1@ecss2 │not running │4m 28s│ │ │ds1@ecss1 │ecss-ds-3.14.10.220 │ds1@ecss1,ds1@ecss2 │ds1@ecss1,ds1@ecss2 │1h 19m│ │ │ds1@ecss2 │ecss-ds-3.14.10.220 │ds1@ecss1,ds1@ecss2 │ds1@ecss1,ds1@ecss2 │4m 28s│ │ │md1@ecss1 │ecss-mediator-3.14.10.220│md1@ecss1,md1@ecss2 │md1@ecss1,md1@ecss2 │1h 19m│ │ │md1@ecss2 │ecss-mediator-3.14.10.220│md1@ecss1,md1@ecss2 │md1@ecss1,md1@ecss2 │4m 28s│ │ │mycelium1@ecss1│ecss-mycelium-3.14.10.220│mycelium1@ecss1,mycelium1@ecss2│not running │1h 19m│ │ │mycelium1@ecss2│ecss-mycelium-3.14.10.220│mycelium1@ecss1,mycelium1@ecss2│not running │4m 28s│ │ │sip1@ecss1 │ecss-pa-sip-3.14.10.220 │sip1@ecss1,sip1@ecss2 │sip1@ecss1,sip1@ecss2│1h 19m│ │ │sip1@ecss2 │ecss-pa-sip-3.14.10.220 │sip1@ecss1,sip1@ecss2 │sip1@ecss1,sip1@ecss2│4m 28s│ └─┴───────────────┴─────────────────────────┴───────────────────────────────┴─────────────────────┴──────┘ All services are started. Active media resource selected list specific: ┌─────────────┬───────┬────────────┬───────────┬───────────┐ │ Node │ MSR │ MSR │ Cc-status │ Cc-uptime │ │ │ │ version │ │ │ ├─────────────┼───────┼────────────┼───────────┼───────────┤ │ core1@ecss1 │ msr_1 │ 3.14.10.67 │ connected │ 01:19:21 │ │ │ msr_2 │ 3.14.10.67 │ connected │ 00:04:23 │ │ core1@ecss2 │ msr_1 │ 3.14.10.67 │ connected │ 00:03:58 │ │ │ msr_2 │ 3.14.10.67 │ connected │ 00:03:56 │ └─────────────┴───────┴────────────┴───────────┴───────────┘
Next:
make sure that the subscribers are registered. Check the number of registered subscribers, it should be approximately the same as at the beginning of the update:
admin@mycelium1@ecss1:/$ domain/bsk.test/sip/user/registered * 320 make users list ... [**********************************************************************] 223mks 320 users information read ... [**********************************************************************] 22ms Executed on the sip1@ecss1 ┌──────────────────┬────────┬─────────┬─────────────────────────────────────────────────────────────────────┬────────────────────────────────────────────────────────────────────┐ │ Interface │ Number │ Dynamic │ Contact │ Registration info │ ├──────────────────┼────────┼─────────┼─────────────────────────────────────────────────────────────────────┼────────────────────────────────────────────────────────────────────┤ ... ┌───────────────┐ │ elements: 320 │ └───────────────┘
- calls go in all directions;
- CFC plays messages;
- Web-conf and other web services are launching;
- no errors are written to the system log. Shell command:
journalctl -f
Uninstalling ECSS components
Removing ecss-mysql
To remove ecss-mysql package while preserving the databases, you can use the command:
sudo apt remove ecss-mysql
At the same time, the databases and configuration files will not be removed.
To completely remove the package along with the databases and configuration files, use the purge key:
sudo apt purge ecss-mysql
Delete confirmation is required for each database. After that, the /var/lib/ecss-mysql/ directory will be cleared.
If you need to completely remove all mysql packages*:
Removing mysql-* packages will also remove ecss-web-conf and ecss-restfs packages.
When removing ecss-restfs, a question is asked about removing all conversation records, and also about removing all MySQL databases.
sudo apt purge mysql-*
Removing ecss-node package
To delete a package, use the command:
sudo apt purge ecss-node
Removing ecss-media-server package
To remove the package while saving the configuration files, use the command:
sudo apt remove ecss-media-server
Configuration files remain in the /etc/ecss/ecss-media-server/ directory.
For complete removal:
sudo apt purge ecss-media-server
Removing unnecessary packages
To remove the remaining packages that are no longer required, use the autoremove key:
sudo apt autoremove
Removing packages by mask
To remove several packages at once, you can use masks (see man apt).
Example of a command to remove all ecss packages:
sudo apt purge ecss-*
ECSS full uninstall commands
To completely remove all ECSS components, run 3 commands:
sudo apt purge ecss-* sudo apt purge mysql-* sudo apt autoremove
After that, restart the server to remove certain processes from RAM.
Logs remain in the /var/log/ecss/security/ directory for service purposes. They cannot be removed by regular means. If you need to clean up this directory, contact technical support.
Debconf
In some cases, instead of reinstalling the package, only a configuration change is required. In this case, use the dpkg-reconfigure <package name> command. Also, if necessary, you can use the commands of the debconf-utils package.