Availability

HAProxy

When the Application Server (AS) must be highly available, a Load Balancer, such as HAProxy, is required. HAProxy is a free, fast, and reliable solution offering high availability, load balancing, and proxying for TCP and HTTP-based applications. The currently supported versions are:

version 1.5: the most feature-wise version; supports SSL, IPv6, keep-alive, DDoS protection, etc.
version 1.4: the most stable version for customers that don't need SSL; it still provides client-side keep-alive.
version 1.3: the old stable version for customers that cannot upgrade for internal policy reasons.

For AS HA testing, version 1.5 was used.

Depending on the complexity of the solution, balancing can be accomplished many ways using different techniques. One of the first things to identify is whether to use Layer 4 or Layer 7 balancing. Each is a solution for different needs, and currently both are used. Note: An Enterprise Edition is also available for HAProxy (see the References section).

Maintenance of an HAProxy deployment is the customer’s responsibility; it is up to the customer to consider either a free version or a commercial version of product.

Application Server

Two Application Servers (AS) are used, each with its own MySQL database.
The databases run master-master replication.
Because of database replication, more than two AS's cannot be used, and only one can be active any one time, while the other AS will be in standby mode.
If configuring multiple AS instances behind a load balancer, you must ensure that the non-host specific configuration entries in the infotypes_saypage.xml file are consistent across the AS nodes.
Also, note that with HA configuration for AS, currently only warm-standby mode can be achieved.

Caveat: If both AS's become active and process requests at the same time, this may cause problems, especially with the database replication. You must ensure that only one AS gets the request at any given time.

MCU

There will be multiple MCUs used in such a system. In this case, all the MCUs will have the HAProxy address configured as the Application Server address (port 80, as internal communication between MCU/NS and AS is HTTP), and therefore will be talking to the active AS always via the proxy. Also, browser clients will use the HAProxy address to connect to the active AS (port 443).

HAProxy Installation and Configuration

Install HAProxy

The HAProxy package is available under the default yum repository for CentOS, Redhat systems. Use the following yum package manager command to install HAProxy on your system:

# yum install haproxy

Or, you can build HAProxy from the source. The source code is covered by GPL v2. Source code and pre-compiled binaries for Linux/x86 can be downloaded here.

Configure HAProxy

The HAProxy configuration file /etc/haproxy/haproxy.cfg must be updated for setting up the service:

# vim /etc/haproxy/haproxy.cfg

In brief:

The HAProxy services ports 80 and 443.
HTTP mode (layer 7) is used for port 80.
TCP mode (layer 4) is used for port 443.
The configuration ‘option forwardfor’ in HAProxy must be enabled for port 80.
- This is for adding X-Forwarded-For header in requests to AS.
The configuration ‘option httpchk’ in HAProxy must be enabled for port 80.
- This is for periodically monitoring the AS's with HTTP requests sent to the URL: /servlet/com.requestec.smg.servlets.Brora?command=get_local_host
- By default, httpchk considers that response statuses 2xx and 3xx are valid, and that other statuses are invalid.
- The default httpchk interval is 2000msec (configurable).
One of the two AS’s is set up as backup, so that this secondary AS will be used only when the primary is down.

A sample configuration file might look like this:

global
# 8 log levels: emerg alert crit err warning notice info debug
log 127.0.0.1   local1 notice
# “Jail Directory” with no write access to anyone.
chroot /usr/share/haproxy
user haproxy
group haproxy
daemon
maxconn 4096
tune.ssl.default-dh-param 2048

defaults
     	log     global
      	mode    tcp
option  tcplog
balance roundrobin
option  logasap
retries 3
maxconn 2000
timeout connect 5000
timeout client  50000
timeout server  50000

# [HTTP Site Configuration]
listen  gvg-as 0.0.0.0:80
# The forwardfor options require mode http.
mode 	http
option	forwardfor
option httpchk GET /servlet/com.requestec.smg.servlets.Brora?command=get_local_host HTTP/1.0
server	gvg-as1 <IP_AS1>:80 check
server	gvg-as2 <IP_AS2>:80 check backup
# The stats require mode http.
Stats	enable
Stats	uri /pstats
stats realm Haproxy\ Statistics
stats auth <admin>:<password>

# [HTTPS Site Configuration]
listen gvg-as-https 0.0.0.0:443
server gvg-as1 <IP_AS1>:443 track gvg-as/gvg-as1
server gvg-as2 <IP_AS2>:443 track gvg-as/gvg-as2 backup

For the above configuration, you might need to do the following:

Create a Jail directory, and remove write permission to anyone:
mkdir –p /usr/share/haproxy
chmod a-w /usr/share/haproxy
If logging is desired, set it up via syslog (which is the only viable option available), by adding the following in the file /etc/rsyslog.d/49-haproxy.con. (This will work if HAProxy is the only service using the UDP syslog port 514, otherwise, put these in /etc/rsyslog.conf instead).
```
#..otherwise consider putting these two in <tt>/etc/rsyslog.conf</tt> instead:
$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

#..and in any case, put these two in <tt>/etc/rsyslog.d/49-haproxy.conf</tt>:

local1.* -/var/log/haproxy_1.log
& ~
#& ~ means not to put what matched in the above line anywhere else for the rest of the rules
#http://serverfault.com/questions/214312/how-to-keep-haproxy-log-messages-out-of-var-log-syslog
```
Now when rsyslog is restarted (‘service rsyslog restart’), logging should go to /var/log/haproxy_1.log (make sure this file gets rotated and/or cleaned as necessary).

Start HAProxy Service

Start HAProxy service using following command, and configure it to auto-start on system boot:

# service haproxy start

# chkconfig haproxy on

Check HAProxy Statistics

The configuration required for the HAProxy Statistics is shown above in the sample configuration file. You should be able to load a browser and connect to the IP of the HAProxy server, on port 80, with the stats URL added to the end:

For example: http://<ha-proxy-address>/pstats

You will be presented with a username/password prompt. Enter the details saved in the haproxy.cfg file that you previously set.

You will be presented with the statistics page similar to this:

HAProxy Redundancy

Important

This is not currently tested.

High availability HAProxy setup is possible with Heartbeat on Linux systems. It requires one floating IP address (which will be used for HAProxy) that will be assigned to one of the computers in the cluster. When the current machine holding the IP address fails, failover server will take the IP address and continue serving requests. This is typically done using Virtual Router Redundancy Protocol (VRRP).

References

MySQL Replication

The MySQL databases on the Active and Passive AS servers can be configured to replicate data between them. This is called a Master/Slave configuration, which can cause problems if the Master is lost. The Slave database will be up-to-date, but once the original Active server is running and ready to act as a failover (Passive) in the cluster, no synchronization of the database will occur. To overcome this problem, the databases should be configured to act as Master/Master, each having a Slave process to the other:

Master2 is the slave to Master1.
Master1 is the slave of Master2.

This will work only if one Master is written to at a time. To do this:

Stop the databases on server1 and server2.
```
service mysqld stop
```
Copy from the server that has the latest /valid data (example, server 1 - to the other server - example server2).
```
scp –r /var/lib/mysql/mzs  server2:/var/lib/mysql
```
On server2 run:
```
chown –R mysql /var/lib/mysql/mzs
```
Followed by:
```
chgrp –R mysql /var/lib/mysql/mzs
```
Restart both MySQL servers:
```
service mysqld start
```
The databases will now be the same.

Step 1

On Master1, make changes in /etc/my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
old_passwords=1

log-bin
# database which should replicated
binlog-do-db=mzs  
# database that should be ignored for replication
binlog-ignore-db=mysql 
server-id=1
slave-skip-errors=1062,1061,1060
[mysql.server]
user=mysql
basedir=/var/lib
[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Step 2

On Master1, create a replication Slave account in MySQL.
Log into the database using the command mysql –u root –p

You will be prompted for the password that you set up during the platform pre-install stage.

mysql> grant replication slave on *.* to 'replication'@<Master2_IP> \
identified by 'slavep1';

Restart the MySQL Master1 by typing service mysqld restart.

Step 3

Edit /etc/my.cnf on Slave1 (Master2).

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
old_passwords=1
server-id=2
#( IP address of remote server in this case Master1)
master-host = <Master1_IP> 
master-user = replication
master-password = slavep1
master-port = 3306
slave-skip-errors=1062,1061,1060
[mysql.server]
user=mysql
basedir=/var/lib

[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Step 4

If there are any problems restarting after these changes due to man page location, on RHEL 4 and 5, add the following link on both servers.
```
cd /var/lib
ln -s /usr/share
```
Restart MySQL Slave1 (Master2) by typing service mysqld restart.
Log in to MySQL and type start slave;
Followed by show slave status \G
Check that Slave_IO_Running = Yes and Slave_SQL_Running = Yes.

Step 5

On Slave1 (Master2), edit /etc/my.cnf and add Master entries into it:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1
server-id=2

master-host = 198.168.16.4 
master-user = replication
master-password = slavep1
master-port = 3306
slave-skip-errors=1062,1061,1060

log-bin                     
#information for becoming master added
binlog-do-db=mzs		 
# information for becoming master added
binlog-ignore-db=mysql      

[mysql.server]
user=mysql
basedir=/var/lib
[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

Step 6

Log into MySQL and create a replication Slave account on Master2 for Master1:

mysql> grant replication slave on *.* to 'replication2'@<Master1_IP> identified by 'slavep2';

Step 7

Edit my.cnf on Master1 for information of its Master.

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
old_passwords=1

log-bin
binlog-do-db=mzs
binlog-ignore-db=mysql

server-id=1
#information for becoming slave.
master-host = <Master2_IP> (IP address of remote server in this case Master 2)
master-user = replication2
master-password = slavep2
master-port = 3306
[mysql.server]
user=mysqlbasedir=/var/lib

Step 8

Restart both MySQL Master1 and Master2.
This can fail with the following error:
```
mysql> start slave;
```
ERROR 1200 (HY000): The server is not configured as slave; fix in config file or with CHANGE MASTER TO
In this case, use the change master to command as shown below, after stopping the slave, and then start it. Note: You may want to first remove the file /var/lib/mysql/master.info.
```
mysql> stop slave;
Query OK, 0 rows affected (0.00 sec)
mysql>  change master to master_host = ‘135.17.37.30’;
Query OK, 0 rows affected (0.01 sec)
mysql> start slave;
```
On MySQL Master2:
```
mysql > show master status;
```
On MySQL Master1:
```
mysql> show slave status\G;
```
Check that Slave_IO_Running = Yes and Slave_SQL_Running = Yes.

Step 9

If the Slave processes are not running, confirm that the replication user can log in remotely from Slave database host into the Master database, for example from <Master1_IP>
```
mysql -u replication –pslavep2 -h <Master2_IP>
```

For more information on MySQL Replication:

Files to be synchronized

Make sure that the following files and directories are synchronized across the two AS servers:

/opt/zenon/sys/login.txt

/opt/zenon/public_html/profiles/

F5 High Availability

When a Collaboration Application Server (AS) must be highly available, similar to HAProxy, an F5 based HA configuration can also be used, where F5 manages the active and standby instance of AS. Any errors detected in the active AS result in a failover to the standby AS and an SNMP alarm.

If configuring multiple Application Server instances behind a load balancer, for non-host specific parameters, please ensure that the non-host specific configuration entries in the infotypes_saypage.xml are consistent across the AS nodes.

F5 HA Design and Configuration

Assume there are two AS boxes: AS1 and AS2 having different IPs [IP1 AND IP2], but the same domain name is used for both.
Initially, both the boxes are set as Passive.
F5 has its own logic to direct the traffic to AS.
Make sure active header is configured in the /opt/zenon/public_html/WEB-INF/infotypes_saypage.xml file.

<ActiveRequestHeader>ha.genrtc.net</ActiveRequestHeader>

F5 sets the above domain name in the Host header. This must be pre-configured because when AS receives traffic, it checks the above IT file that it has those headers set.

When AS writes active headers and IP to the database table, the header must match with the domain names in the database table. The box that receive traffic sets itself to ACTIVE and writes it to a database table called maxmcu_active_zas.

The table stores the ACTIVE box IP and the domain name along with the updated timestamp. Make sure that the above table exists. There is no need to pre-populate the table as AS clears the table on startup and populates it when F5 traffic is received.

Both boxes have their own set of databases: DB1 and DB2. Data is synchronized between two databases, using mysql replication.
Each box has a monitor running, internal to each AS. This monitor periodically checks if the LOCALHOST IP of the current box is equal to that stored inside maxmcu_active_zas table. If that’s not the case, it sets itself to PASSIVE.
On the Passive box, the monitor returns immediately and doesn't check the database. Sleep time, in milliseconds, of this thread can be configured using this IT:

<ActiveZasMonitorSleepTime>1000</ActiveZasMonitorSleepTime>

F5 periodically keeps checking and tries a ping command – get_local_host. If F5 doesn't get a response, F5 redirects traffic to the other box.

URL Example:

http://rbw1.genrtc.net/servlet/com.requestec.smg.servlets.Brora?command=get_local_host

Response:

PRODUCTION ACTIVE
10.144.245.74

N+M Instances of Collaboration MCU

In general, if Genesys Video Gateway is deployed as a farm of N MCU instances and you want to prepare for the potential failure of M instances, you can maintain maximum availability of the service by deploying N + M MCU instances. Configure the AS to detect the failure of any MCU instances and to direct incoming sessions to the remaining instances.

Contents