Jump to: navigation, search

Data Migration from iWD 8.5.1x to iWD 9.0

There are two different options for data migration:

  • Migrating GTL History Events—transfer GTL history data to iWD Manager 9.0.
  • Migrating Data Mart History Events—transfer iWD task data to iWD Data Mart 9.0.
Important
  1. Data migration is not a mandatory step. Depending on your needs you can choose one option, both or none.
  2. The migration process might be intensive and time consuming. It is recommended to avoid migration unless absolutely necessary.
  3. If you cannot avoid the GTL History migration, please consider using the taskEventLimit option to reduce the amount of data to be migrated.

Please also note that depending on the selected iWD History Node configuration, IF:

  • The option [iWD]/process-gtl = false, there is no point in migrating GTL History Events.
  • The option [iWD]/process-dm = false, there is no point in migrating Data Mart History Events.

Migrating GTL History Events

The purpose of this migration is to transfer task history data to iWD Manager 9.0.

You can find Task History by clicking the History tab in the GTL view. It displays all the task attributes’ values and the task history events.

Limitations

Task history migration is supported for tasks that are still in the Interaction Server database.

Prerequisites

  1. Installing iWD History Node is done.
  2. iWD History Node Configuration is done
  3. Stop all iWD 8.5 components.
    1. iWD Manager's Tomcat.
    2. iWD Runtime Node.
    3. iWD Web.
    4. Interaction Server.
Warning
If Interaction Server is not stopped, no iWD task activity during the migration process will be recorded.

Procedure

You can migrate GTL history events from the Event Log database to the History Node database by using History Node's built-in mechanism, which can be run with the following command:

On Linux

./iwd_history.sh -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP events-migrate <yaml-config-filename>

On Windows

JavaServerStarter.exe -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP events-migrate <yaml-config-filename>

where <yaml-config-filename> is the path to the YAML configuration file providing connection settings to the old environment.


Warning
When events migration is in process, make sure that one of the following conditions applies:
  • No History Node applications are running.
  • If any History Node applications are running, they must have history events removal disabled (that is, the stop-processing option must be set to keep)
  • The GTL History Events Migration mechanism only migrates events into the iWD Manager database and doesn't support migration to the Data Mart database.

Prepare YAML Configuration File

  1. Please find the configuration template file according to your database vendor:
    Template path
    ${INSTALLATION DIRECTORY}/config/template_iwd_events_migrate_<DB-VENDOR>.yaml.template
  2. Copy and rename.
  3. Edit the template and set the following parameters:
    • solutionId: <solution-id>
    • taskEventLimit: <number>
    • interaction-server:
      • url: <url>
      • user: <user>
      • password: <password>
    • eventlog:
      • url: <url>
      • user: <user>
      • password: <password>
Important
For security reasons, delete the passwords once migration/creation is complete.

Configuration Options

  • logging—Defines History Node logging configuration.
  • solutionId—The runtime id of the solution for which events will be migrated. Solution with this id must be configured in Configuration Server.
  • taskEventLimit—Allows to specify the maximum number of events to be migrated for one iWD task.
    • If a task consists of more events than specified, only the last of them will be migrated. It might be helpful for stability and performance reasons.
    • All the tasks with skipped events will be displayed in the migration log.
      • Valid values:
        0—No limit (default)
        1 to 2147483647—The maximum number of events
        Important
        iWD Manager cannot show more than 5000 events per one interaction. If you don't have any other specific needs then 5000 is the recommended value.
  • interaction-server—Points to the Interaction Server database.
  • eventlog—Points to the Event Log database.

Working Command Line with Output Example

Output Example 

INFO [2020-06-22 17:03:02,104] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50011|Command description: Migrate events from Event Log db to the History Node
INFO [2020-06-22 17:03:10,229] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50012|Connecting to the Interaction Server database
INFO [2020-06-22 17:03:11,979] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50013|Events for 74480 interaction(s) will be migrated
INFO [2020-06-22 17:03:11,979] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50014|Connecting to the Event Log database
WARN [2020-06-22 17:03:12,925] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50071|Events limit exceeded, interaction: 01T6XTJXMYTMNEFR
INFO [2020-06-22 17:03:54,544] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50016|Migration progress: 7448/74480
...
INFO [2020-06-22 17:08:14,220] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50016|Migration progress: 67032/74480
WARN [2020-06-22 17:08:44,247] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50071|Events limit exceeded, interaction: 02RHK42XSWDA4009
INFO [2020-06-22 17:08:45,247] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50016|Migration progress: 74480/74480
INFO [2020-06-22 17:08:45,247] com.genesyslab.iwd.eventlog.EventLogEventsMigrator: #50015|Migration finished
INFO [2020-06-22 17:08:45,263] com.genesyslab.iwd.history.HistoryApplication: #50047|Shutdown hook triggered
INFO [2020-06-22 17:08:45,263] com.genesyslab.iwd.history.HistoryApplication: #50048|Shutdown hook completed
info
Important
Message #50071 shows interactions with skipped events due to specified taskEventLimit option.


Migrating iWD Data Mart History Events

The purpose of this migration is to transfer the current tasks data to the iWD Data Mart 9.0. This way you will be able to see that data in iWD 9.0 reporting with some limitations (see below).

The procedure consists of two steps:

  1. Migration from the EventLog database to iWD History Node.
  2. Loading the data from iWD History Node to iWD Data Mart.

Limitations

  • Only unfinished tasks are migrated (backlog).
  • The whole task event history is not migrated.
    • Included:
      • Agent's activity events: ASSIGN, FINISH, FINISH_RETURN, TRANSFER
      • A task "snapshot" or the latest task state - migrated as NEW
    • Excluded:

Step 1: Migration from EventLog Database to iWD History Node

Prerequisites

  1. Installing iWD History Node is done.
  2. iWD History Node Configuration is done.
  3. Stop all iWD 8.5 components.
    1. iWD Manager's Tomcat.
    2. iWD Runtime Node.
    3. iWD Web.
    4. Interaction Server.
Warning
If the Interaction Server is not stopped, all iWD tasks activity during the migration process will not be recorded.

Procedure

You can migrate Data Mart history events from the Event Log database to the History Node database by using History Node's built-in mechanism, which can be run with the following command: On Linux 

./iwd_history.sh -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP datamart-migrate <yaml-config-filename>

On Windows 

JavaServerStarter.exe -host CFG_SRV_HOST -port CFG_SRV_PORT -app HN_APP datamart-migrate <yaml-config-filename>

Where <yaml-config-filename> is the path to the YAML configuration file providing connection settings to the old environment.

Prepare YAML Configuration File

  1. Please find the configuration template file according to your database vendor:
    Template path
    ${INSTALLATION DIRECTORY}/config/template_iwd_datamart_migrate_<DB-VENDOR>.yaml.template
  2. Copy and rename.
  3. Please modify the following options:
    • solutionID: <solution-id>
    • fetchSize: 100
      • customAttributes:
        - <attribute1>
        - ...
        - <attribute10>
      • interaction-server:
        <url>
        <user>
        <password>
      • eventlog:
        <url>
        <user>
        <password>
Warning
Take care to set the customAttributes section properly, otherwise they won't be migrated.

Configuration Options

  • logging—Defines History Node logging configuration.
  • solutionId—The runtime id of the solution for which events will be migrated. Solution with this id must be configured in Configuration Server.
  • fetchSize—Determines how many interactions are processed per batch.
    This option in combination with processor-threads could be useful for performance and stability tuning.
    • Valid values:
      • 1 to 2147483647 (100 by default)
  • customAttributes—Defines a list of the custom attributes.
    Important
    You should provide the full list of Task Attributes from iWD 8.5. Attributes not listed in this field will not be migrated.
  • interaction-server—Points to the Interaction Server database.
  • eventlog—Points to the Event Log database.

Working Command Line with Output Example

INFO  [2020-06-10 16:40:22,668] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50011|Command description: Migrate events from Event Log db to the Data Mart
INFO  [2020-06-10 16:40:32,356] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50012|Connecting to the Interaction Server database
INFO  [2020-06-10 16:40:33,496] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50013|Events for 71320 interaction(s) will be migrated
INFO  [2020-06-10 16:40:33,496] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50014|Connecting to the Event Log database
INFO  [2020-06-10 16:40:39,584] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50016|Migration progress: 10/71320
...
INFO  [2020-06-10 17:42:29,923] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50016|Migration progress: 71320/71320
INFO  [2020-06-10 17:42:29,923] com.genesyslab.iwd.eventlog.EventLogData MartMigrator: #50015|Migration finished
Important
Message '#50016 Migration progress' shows the current migration progress.

Any errors are logged. For more detailed information you can use DEBUG or TRACE log level.

Step 2: Loading the data from iWD History Node to iWD Data Mart

Prerequisites

  1. Step 1 Migration from EventLog DB to iWD History Node has been completed.
  2. The Installing Runtime Node is done.
  3. The iWD Runtime Node Options are configured.
  4. The iWD Aggregate Plugins are configured.

Procedure

  1. Configure the specific migration options
    1. Open the iWD Data Mart Application Properties: Configuration → iWD Datamart Properties → options
    2. Add a new option [settings]\etl-start-date.
      • Description: the earliest date for which iWD Data Mart initialize time data in a new deployment.
        Valid values: date in format yyyy-MM-dd HH:mm, like 2015-10-26 00:00
        Default value: the beginning of the current year.
        Note: Perform the following query to the Interaction Server Database in order to know the proper value. 
        SELECT min(submitted_at) as ETL_START_DATE
FROM interactions
    3. Add a new option settings\aggregate-start-date
      • Description: the earliest date for which iWD Data Mart aggregates data in a new deployment.
        Valid values: date in format yyyy-MM-dd HH:mm, like 2015-10-26 00:00
        Should be greater than etl-start-date.
        Default value: today - 30 days
        Warning
        If the etl-start-date or aggregate-start-date options are not set (or set inappropriately), it may cause "orphaned" rows in facts and aggregates."Orphaned" rows means that such records don't have a link to the DATE_TIME dimension and cannot be selected or filtered by date.
    4. Add a new option [schedules]\manual-mode = true.
      • Description: It switches off a scheduled mode and allows to start ETL jobs manually
  2. Start the Runtime Node 9.0.
    1. Open Data Mart Dashboard—A real-time view of the status of iWD Services.
    2. Run ETL jobs manually in the following pattern:
      1. Run Initialize job
      2. Run Load Config
      3. Run Load Intraday Job
      4. Run Aggregate Intraday Job
        Important
        Aggregate Intraday Job can process only 1 DAY of data per run. In order to catch up faster you should repeat this job as many times as needed until all the data is aggregated.
      5. Run Load Historical Job
      6. Run Aggregate Historical Job
    3. Make sure that all the data was loaded
      1. The iWD History Node datamart table is empty. 
        SELECT count(*) cnt_rows
FROM <HISTORY_NODE_DB>.datamart
      2. The iWD Data Mart loaded data without fails. 
        SELECT *
FROM <DATAMART_DB>.etl_audit
WHERE STATUS in ('IN PROGRESS', 'FAILED')
  3. Rollback the specific migration options:
    1. Remove [settings]\etl-start-date.
    2. Remove [settings]\aggregate-start-date.
    3. Remove [schedules]\manual-mode.
This page was last edited on June 30, 2020, at 17:00.
Comments or questions about this documentation? Contact us for support!