Santhosh Annamalai

1. Install 64-bit Python 3.11.5 from the Python website.
2. Install Recording Muxer Script IP with the installer.

Note: Install the following third-party libraries in the order they appear and untar the files in Administrator mode.

3. Untar the <Recording Muxer Install Directory>/thirdparty/jmespath-1.0.1.tar.gz file.
4. From the <Recording Muxer Install Directory>/thirdparty/jmespath-1.0.1 directory, run py -m pip install . . in a command line interface.
5. Untar the <Recording Muxer Install Directory>/thirdparty/urllib3-1.26.16.tar.gz file.
6. From the <Recording Muxer Install Directory>/thirdparty/urllib3-1.26.16 directory, run:

   py -m pip install . .

7. Untar the <Recording Muxer Install Directory>/thirdparty/six-1.16.0.tar.gz file.
8. Run py -m pip install . . from the <Recording Muxer Install Directory>/thirdparty/six-1.16.0 directory.
9. Untar the <Recording Muxer Install Directory>/thirdparty/python-dateutil-2.8.2.tar.gz file.

Important

Starting from (version placeholder), ORS supports using Redis for session persistence and recovery over a secure connection. For more information, see ORS with Redis Cluster.

type

Option section: persistence

Configuration object: ORS Application object

Default value: cassandra

Valid values: cassandra, redis

Value changes: take effect during startup. Changes to this option are not applied dynamically.

This option specifies what type of session recovery must be used. Only supported types are Cassandra and Redis Cluster. If the option value is different from redis, ORS will try connect to Cassandra. Changing this option after startup will not have an effect.

redis-nodes

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values: A list of host names and ports for redis nodes in the format of <HOST>:<PORT;[<HOST>:<PORT>...]

Value changes: take effect during startup. Changes to this option are not applied dynamically.

This option provides hosts and corresponding ports when using the Redis Cluster based persistence type. This is a semi-colon separated list of host and port of Redis Cluster nodes in the format, <HOST>:<PORT>. A host can be a hostname or an IP address.

For example: persistence/redis-nodes = rhost:6379;143.244.220.15:8301;test47:1234. It is sufficient to configure a single alive node for ORS to connect to all cluster nodes.

enable-multimedia-persistence

Option section: persistence

Configuration object: ORS Application object

Default value: cassandra

Valid values: true, false

Value changes: take effect during startup. Changes to this option are not applied dynamically.

This option specifies if persistence must be enabled for multimedia interactions. Set to true to enable persistence for both Redis and Cassandra. Set to false to disable persistence for multimedia interactions regardless of whether Redis or Cassandra is used.

username

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

Specifies the username that is used to authenticate with Cassandra or Redis Cluster.

password

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

Specifies the password that is used to authenticate with Cassandra or Redis Cluster.

redis-connect-timeout

Option section: persistence

Configuration object: ORS Application object

Default value: 100

Valid values: Any integer value

Value changes: take effect during startup. Changes to this option are not applied dynamically.

Specifies the timeout in milliseconds, that Redis++ client library will use for establishing connection. It is important to configure this timeout as a small value to avoid blocking ORS worker thread execution if Redis becomes unavailable and reconnection is attempted as part of executing the Redis request.

If set to 0, the timeout is disabled.

redis-socket-timeout

Option section: persistence

Configuration object: ORS Application object

Default value: 100

Valid values: Any integer value

Value changes: take effect during startup. Changes to this option are not applied dynamically.

ORS communicates with Redis synchronously. This option defines the timeout in milliseconds, Redis++ client library will use to send read or write request to Redis. It is important to configure this timeout as a small value to avoid blocking ORS worker thread execution if Redis becomes unavailable.

Note: Actual max time for execution of Redis request can exceed this option value.

If set to 0, request to Redis will never timeout and block ORS worker thread until completed successfully or get an error.

tls-enabled

Option section: persistence

Configuration object: ORS Application object

Default value: false

Valid values: true,false

Value changes: take effect during startup. Changes to this option are not applied dynamically.

If set to true, connections to the Redis instances will be encrypted. Note: TLS is not supported on Windows platform. For details regarding Redis TLS(SSL), refer to Redis documentation, https://redis.io/docs/connect/cli/ .

tls-cert

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

If the option tls-enabled is set to true, you can specify in this option a file name of TLS certificate. Example: /path/to/client/certificate

tls-key

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

If the option tls-enabled is set to true, you can specify in this option a corresponding file name of TLS private key to the certificate configured in tls-cert. Example: /path/to/private/key/file

tls-cacert

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

If the option tls-enabled is set to true, you can configure a trusted root certificate bundle using this option. Example: /path/to/CA/certificate/file

tls-cacertdir

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

If the option tls-enabled is set to true, you can configure a trusted root certificate directory instead of tls-cacert.

tls-sni

Option section: persistence

Configuration object: ORS Application object

Default value:

Valid values:

Value changes: take effect during startup. Changes to this option are not applied dynamically.

If the option tls-enabled is set to true, you can specify (optional) in this option, a Server Name Indicator (SNI) value.

=ORS with Redis Cluster - Page under development. Do no publish!=

Overview

ORS now supports using Redis for session persistence and recovery over a secure connection.

Redis is an open source (BSD licensed), in-memory data structure store used as a database, cache, message broker, and streaming engine. To learn more about Redis, visit https://redis.io/docs/about/ .

Capacity

Redis recommended capacity estimate Number of Redis Cluster Nodes and the amount of memory are vary much depend on number of expected simultaneous ORS sessions on the ORS Cluster and session complexity. We recommend to use Redis Cluster with at least 3 nodes. On average 3 nodes Redis Cluster should be sufficient to handle 9 HA pairs of ORS applications.

Migration steps

The migration from C* to Redis can be done considering following scenarios:

Preparation phase

1. Have Redis Cluster deployed and configured for ORS persistence. Deploy Redis Cluster CPU and memory monitoring tools
2. Upgrade ORS applications to the latest recommended version,
3. Configure all necessary ORS Application options for persistence to Redis, but not enable it (don't set [persistence]type="redis" yet)
4. Restart ORSes using standard procedure (restart backups, switchover, restart new backups)
5. Verify ORS applications continue to work as expected. Verify C* persistence work as expected if enabled

Enable Redis persistence

1. Enable Redis persistence ([persistence]type="redis" ) for backup applications. Restart backup applications for new setting to take effect. Trigger switchower to make them Primary.
2. Verify Primary ORS applications now connected and doing serialization to Redis. After switchover ,ORS sessions which were in progress, will not be restored and routing may failed for those calls. Verify that new ORS sessions are processed as expected.
3. Monitor Redis Cluster nodes CPU and memory consumption. Add more Redis nodes or upgrade memory capacity depending on monitoring data. Consult Redis support if necessary.
4. Enable Redis persistence ([persistence]type="redis" ) for backup applications.. Restart backup applications for new setting to take effect. Trigger switchower to make them Primary.
5. Verify new Primary ORS applications now connected and doing serialization to Redis. After switchover, ORS sessions which were in progress, should be restored from Redis. Verify that all calls processed as expected.

Rollback

1. If new solution doesn't work - disable Redis persistence for all ORS applications.
2. Restart ORSes using standard procedure (restart backups, switchover, restart new backups)
3. Investigate the reason new solution doesn't work as expected

Finalize phase

Remove Cassandra when you feel comfortable with new Redis solution

Configuration

Configuration Options

name	section	default value	Value changes	Valid values	Description
type	persistence	cassandra	During startup	cassandra, redis	Determine what type of session recovery we plan to use. Only supported types C* and Redis Cluster. If option value is different from redis, ORS will try connect to C*. Changing this option after startup will not have effect
redis-nodes	persistence		During startup	<HOST>:<PORT;[<HOST>:<PORT>...]	This option provides hosts and correspondent ports when using the Redis Cluster based persistence type. This is a semi-colon separated list of of host and port of Redis Cluster nodes in format <HOST>:<PORT>. Host can be host name or Ip Address. For example: persistence/redis-nodes = rhost:6379;143.244.220.15:8301;test47:1234. It's sufficient to configure single alive node for ORS to connect to all cluster nodes
enable-multimedia-persistence	persistence	cassandra	During startup	true, false	This is hidden option specifies if persistence will be enabled for multimedia interactions depending on type of persistence. true means that persistence is enabled for both Redis and Cassandra, false - disabled for all persistence types.
username	persistence		During startup		The username used to authenticate with Cassandra or Redis Cluster
password	persistence		During startup		The password used to authenticate with Cassandra or Redis Cluster
tls-enabled	persistence	false	During startup	true, false	If set to "true" connections to the Redis instances will be encrypted. TLS is not supported on Windows platform For details regarding Reid TLS(SSL) please refer to: Redis-cli SSL documentation
tls-cert	persistence		During startup		If option tls-enabled set to true, here you can specify a file name of TLS certificate. Example "/path/to/client/certificate"
tls-key	persistence		During startup		If option tls-enabled set to true, you can specify a corresponding file name of TLS private key to a certificate configured in tls-cert. Example "/path/to/private/key/file"
tls-cacert	persistence		During startup		If option tls-enabled set to true, you can configure a trusted root certificate bundle. Example: "/path/to/CA/certificate/file"
tls-cacertdir	persistence		During startup		If option tls-enabled set to true, you can configure a trusted root certificate directory instead of tls-cacert
tls-sni	persistence		During startup		if option tls-enabled set to true, here optionally you can specify Server Name Indicator (SNI) value.

Design ORS with Redis Cluster

Redis scales horizontally with a Redis Cluster topology. Data is automatically shared across multiple Redis Cluster nodes. Redis Cluster also provides some degree of availability.

ORS will only support persistence with Redis Cluster ( will be no support for connection to Redis Sentinel or standalone Redis instance).

Pull of connections to Redis Cluster will be shared between ORS worker threads - consequently each working thread will have TCP connections to each Redis Cluster node

Disable persistence and recovery of digital sessions

Generally, we never recommend persistence and recovery of digital sessions. In case of ORS switchover, failover or disconnect, digital interactions are immediately placed back in queue by eServices. This is the major different from voice interactions, Therefore, there is no need to persist and recover digital sessions. When connection between ORS and IXN service is restored, interactions will continue to be processed according interaction queue logic. Even for chat interactions, there is no need in session recovery, as chat logic is not controlled by ORS session. There is no risk of incomplete or stuck session on ORS due to restoration or missing the event. Persistence of digital session puts performance penalty on ORS applications and Cassandra or Redis. Also persistent data for digital sessions put extra capacity requirement for Cassandra or Redis in-memory data storage.

Therefore, persistence and recovery of digital sessions triggered by eServices interaction to Redis is disabled.

Mapping C* persistence schema to Redis API

C* Column family effectively is keyed set of records, with every record being set of Columns (that is set of key-value pairs). As result, straightforward conversion to Redis is presenting every single Cassandra record from any ColumnFamily with Redis Hash object (i.e. set of key-value pairs) with name like <ColumnFamilyName>:<RecordKeyInsideThisColumnFamily>.

The Redis Plus Plus library will be used for communication with Redis Cluster. The Reids interface will be used exclusively, there will be no option to use old interface with C* . The SSL/TLS features available in the library will be employed to implement Orchestration support for TLS encrypted connections to the Redis cluster.

ORS will store data in Redis under following key <BASE> = ORS:<ORS_CLUSTER_NAME>

All ORS data will have TTL set to prevent stuck Redis entries and Redis DB memory grow

Below is proposition about C* replacement with Redis.

Mapping table

Expiration and cleanup of data stored in Redis

Redis works with an in-memory dataset. ORS deletes all keys it store in Redis when they not needed. Still we need to exclude any possibility to potentially have unneeded data accumulated there. Therefore each record we add to Redis should either have expire time (TTL) associated or periodical cleaning up procedure executed

Where it's possible, we setting expire time to data added to Redis as described in above table.

For SessionIDServerInfoRIndex data. This is a key value pair stored in Redis, where key is nodeId and the value is a list of sessionIds associated with that ORS instance. This list gets updated every time new session is created or old one is deleted. In order to oversee this list doesn't have obsolete items, ORS Application, with that nodeId, will reiterate through the list and check if associated session is stored in Redis or present in ORS memory and remove orphan sessionIds. Check will be done every 24 hours

For RecoverSessionIDServerInfoRIndex data. This is a key value pair stored in Redis, where key is nodeId and the value is a list of sessionIds marked for proactive recovery. This list will be cleaned upon ORS Application Startup

Metrics

N	Metric	HTTP stat	Description
1	ors_redis_doc_stored	redisDocStored	number of times SCXML documents are stored to Redis
2	ors_redis_doc_removed	redisDocRemoved	number of times SCXML documents is removed from Redis
3	ors_redis_doc_read	redisDocRead	number of times SCXML documents is read from Redis
4	ors_redis_doc_data_amount	redisDocDataAmount	amount of SCXML documents data in stored to Redis in Bytes
5	ors_redis_session_stored	redisSessionStored	number of times Redis Sessions are stored
6	ors_redis_session_removed	redisSessionRemoved	number of times Redis Sessions are removed from Redis
7	ors_redis_session_read	redisSessionRead	number of times Redis Sessions are read from Redis
8	ors_redis_session_data_amount	redisSessionDataAmount	amount of session data stored to Redis in Bytes
9	ors_redis_writes	redisWrites	total number of writes operations with Redis
10	ors_redis_deletes	redisDeletes	total number of delete operations with Redis
11	ors_redis_reads	redisReads	total number of read operations to Redis
12	ors_redis_errors	redisErrors	Number of Redis operations completed with error
13	ors_redis_time_le100	redisTimeLess100	Number of Redis operations with response time less than 100 ms
14	ors_redis_time_le300	redisTime100to300	Number of Redis operations with response time within 100-300 ms
15	ors_redis_time_le1200	redisTime300to1200	Number of Redis operations with response time within 300-1200 ms
16	ors_redis_time_le5000	redisTime1200to5sec	Number of Redis operations with response time within 1.2-5 s
17	ors_redis_time_gt5000	redisTimeMore5sec	Number of Redis operations with response time more than 5 s