A/B Model Performance Testing

Predictive Routing enables you to use A/B testing to assess how well your Predictive Routing models perform when compared with other models or with your standard routing.

• Learn more about models.

When you run an A/B test, you compare the outcome of routing using one model with the outcome from one or more other models that differ from the first in specific, targeted ways. After a set amount of time, an analysis of the outcomes can show you the strengths and weaknesses of each model.

A/B Testing

A/B testing is a variety of testing that involves single element changes across multiple variations. Predictive Routing enables you to test various models against each other with respect to a specific metric. For example, you might be able to evaluate which model best achieves an improvement in first contact resolution, compared to each other and to routing without Predictive Routing.

Use A/B Test Results for Contact Center Wide Decisions

You can use A/B testing to evaluate the impact of changes both models and as well as for the impact of other routing decisions. Such tests help you see what elements are important for your agents and customers in a controlled environment. Once you have seen the results of a test you can apply the new knowledge to your interaction routing design or training plans.

Advantages of A/B Testing

• You can see results quickly.
• You can use it to test outcomes from advanced analytics, such as the effect on handle time, speed of answer, transfers, and so on.
• Uncovers features you can manipulate to make significant improvements to Key Performance Indicators (KPIs).
• Helps to prove that predictive routing improves targeted KPIs compared with skill-based routing.
• Requires a relatively small number of interactions.

How Many Interactions Do You Need for A/B Testing?

A common question is how many interactions you need when running a test. This depends of course on a number of factors. You might be able to make useful decisions based on only 100-200 interactions, especially when you have some scenarios where some aspect of your environment is significantly underperforming. In general, you should allocate at least 1,000 total interactions to your test stream and run the test for at least a week to account for any daily variances.

How Time-Sliced A/B Testing Works

Predictive Routing provides time-sliced A/B testing, with which you can enable and then disable Predictive Routing for specific periods of time. You can compare results from a Predictive Routing model and routing without Predictive Routing. These time periods should be long enough to allow for a large enough sample size for each time period. Genesys recommends periods of at least 3 times your maximum speed of answer to give enough samples in each period.

Important

To use time-sliced A/B testing, all interactions must come from queues using Predictive Routing.

To be useful, these time periods also need to be comparable in expected volume of interactions, and number and quality of agents.

To configure A/B testing:

• Set the value of the prr-mode configuration option to ab-test-time-sliced.
• Configure the length in seconds of the time-slices that go into each sub-stream in the ab-test-time-slice configuration option.

Getting Started with A/B Testing

The features or steps in your current targeting strategy, such as the skill expression used, agent skill sets, training plans, shifts worked, and so on, have consequences. Those consequences are benefits that have been designed to suit your business and operational model. When you consider how to use A/B testing to improve your results, keep in mind the following principles:

• Focus on a specific KPI or metric that you can pin-point for testing.
• Make sure you are aiming to improve a metric that matters.
• Study the data to identify as clearly as possible which features affect that KPI. The testing itself then enables you refine your judgment.
• Clearly identify which features are necessarily static given your environment and which you can change.
• Change one feature at a time when fine-tuning your models, so you can clearly identify which have real impact.

Building a Strategy, Predictor, and Models

To start testing, create a main strategy. Insert the ActivatePredictiveRouting subroutine into the appropriate place in the strategy flow, then use the Predictive Routing interface to create a predictor for the strategy. For each predictor, you can create multiple models. By activating and deactivating one or more models from the Predictive Routing interface, you can control your testing without having to edit your strategy each time.

Predictive Routing will work if you are targeting all agents in your contact center, agent groups, or personalized direct-to-agent targeting.

Building Predictors

Each predictor can be built to operate on only one metric, such as first contact resolution, NPS, handle time, revenue, and so on. You might want to test predictors against each other, to see which metric gets the most lift from Predictive Routing, but for the most part, you will be testing models against models, not predictors against predictors.

Building Models

A model is a variant of a predictor, which you configure to use a specific set of agent and customer features. By changing one feature at a time in various models, you can focus in on the features with the greatest impact on results. Start with the default model created by Predictive Routing, which includes all customer and agent data, then analyze and adjust from there.

Important

For instructions on how to create and activate models, see Configuring, Training, and Testing Models, which you can access from this link or directly from the interface by clicking the ? icon.

Test Mode

Predictive Routing has various modes, which can be set using the prr-mode configuration option. You can also configure a Switch block in your routing strategy, which can direct interactions along paths configured with the various modes:

• Dry-run—Predictive Routing scores agents for your interactions, but does not use the scores for routing.
• A/B time-sliced
• Production—Predictive Routing scores are used to route all interactions.

Test mode is especially useful when you are doing early-stage model analysis to identify the most promising models for additional A/B testing.

Pitfalls to Avoid

• You are testing a hypothesis that is not based on your data.
• Your data does not contain the information you need to test the metric you have selected.
• You are using different agent pools, agent groups, or target skills for A/B testing, which means you have multiple variables operating at once.
• Your models are running in dissimilar strategies.
• You are using time-slicing testing but your time period is too short for significant results.