Configuring, Training, and Testing Models
A Model is built on a Predictor and includes the same target metric. Each Model has a subset of the agent and customer features present in the Dataset. The Feature Analysis report helps you to identify the features with the strongest impact on the target metric. You can create multiple Models for the same Predictor, each with a different set of features selected.
- You can compare how well Models work to create the most effective ones.
- You can configure Models that are best-suited to specific circumstances and control which Model is used by activating and deactivating them. This enables you to respond promptly to changes such as weekday vs weekend volume or the anticipated increase in certain types of interactions after a big marketing push.
When you create a Predictor, a full feature set Model is created automatically, including all the agent and customer features populated from the Predictor. A Predictor can have a number of Models associated with it.
- The right-hand toggle navigation menu enables you to view a tree view of all Datasets associated with your Account, with the Predictors and Models configured for each. To open or close this navigation menu, click the icon.
- You must reload the page to view updates made using the Predictive Routing API, such as appending data to a Dataset, creating, updating, or deleting a Predictor, or creating, updating, or deleting a Model.
The full feature set Model requires a complete set of Local Models, one for each agent. If it's not possible to train a Local Model for a specific agent, GPR generates the following log message, indicating that the Local Model has been skipped for a specific agent, and does not create the full feature set Model: WARNING <BOTTLE> models.py:420 RETRAIN: Skipping training for individual model <agent_id> since no data is available for classes.
For example, you might have a Boolean metric, such as FCR. If, for a specific agent, all entries from the training set had only examples of one Boolean class (that is, either all were resolved, or none were resolved), GPR cannot create a Local Model for that agent.
- You can configure a routing strategy to use a specific Predictor, then edit the Models and change which are active. In this way, routing can be adjusted and optimized on the fly, without requiring you to edit the strategy. For instructions on this specific functionality, see Activating multiple Models at once, below.
- The list of Models includes the Quality column, which provides analysis reports on Model Quality and Agent Coverage.
Creating and Configuring a New Model
To open the configuration menu, click the Settings gear icon, located on the right side of the top menu bar: .
Procedure: Open the Models interface
To access your Models:
- Select Predictor from the left-hand navigation bar and then click the name of a Predictor in your list.
- The Predictor configuration window opens.
- Click Models.
Procedure: Add a new Model
When you create a Predictor, GPR automatically adds the full feature set model, as shown in this graphic. The full feature set model includes every agent and customer feature you selected when you created the Predictor.
- To create a new Model, click Add Predictor Model.
Procedure: Configure a Model
Edit the fields as explained below.
- To choose Agent Features and Customer Features, click in the appropriate text box and select the desired features from the drop-down list.
- Agent Features are items that refer to the agent. All agent-related fields in your selected Dataset appear in the Agent Features field.
- Customer Features are items that refer to the customer or that are available in interaction user data. They refer to aspects of the environment, broadly speaking, in which the interaction occurs.
You can only select from the Agent Features and Customer Features that are included in the Predictor.
- To remove a feature, click the X in the box containing the feature name.
- Choose one of the Model types, GLOBAL, DISJOINT, or HYBRID.
- A single Model is built to predict agent scores.
- Provides generalizations that are probably true across the whole pool of agents. For example, agents that are part of this group might have a lower transfer rate; agents with longer tenure generally have higher resolution performance; and so on.
- Select this Model type only if there are action features that correlate with the target metric.
- In the absence of Agent Features, the Global Model type produces the same score for every agent, which makes it useless for ranking.
- A Model, also called a Local Model, is built for each agent.
- Captures agent-specific idiosyncrasies with respect to performance.
- With this option, a Global Model is also built to produce scores for agents for whom individual Models cannot be created. For example, some agents might not have enough data to create an individual Model.
- Combination of Global and Disjoint Models, which uses an average of their scores.
- Can both provide generalization and capture agent-specific nuances in performance.
- Select this Model only if there are Agent Features that correlate with the target metric; that is, the metric (first-contact resolution, net promoter score, average handle time, and so on) for which the associated Predictor is built.
Procedure: Set the training and testing percentages
You can configure how much of your Dataset is used to train your Model and how much is used to test how well it works. This split is time-based. The most recent interactions are allocated to the test section of the Dataset. For example, if you use 80% of the data to train your Model and 20% to test it, the most recent 20% of the Dataset records are used for testing.
To set this value:
- Use your mouse to slide the indicator to the desired point on the Train/Test bar.
Procedure: Train your new Model
After you create your Model, you must train it on your data. When a trained Model has not been yet activated, you can modify it. For example, you might add new Agent or Customer Feature, change the train/test split, and so on. After those modifications, you must retrain your Model. When you train or retrain a Model, the integer in the Version column of the list of Models is incremented.
If you change the date range for the Predictor data, then purge and regenerate your dataset, an already-trained Model does not need to be retrained. It uses the previously configured date range. But all Models created after the data purge and regeneration use the new date range.
- Select the check box in the table row for your Model.
- Click Train.
The Training job can take a fairly long time, depending on the size of your Dataset. Click the Jobs tab to monitor job progress.
Viewing Agent Coverage and Model Quality Reports
After you train your Model, the Quality column shows values for various methods of evaluating how well the Model works. The evaluation methods are selected automatically depending on the type of Model.
Procedure: View Model Quality and Agent Coverage reports
GPR provides the following analysis types:
- Classification analysis buckets observations into predetermined categories, based on data already used for training. In this case, you already know that the data can be divided into meaningful categories, into which your new data can be placed. For example, you might record interaction results showing whether a customer's issue was resolved or not; whether the desired AHT was met or not; whether the final NPS was above a certain value or not; and so on.
- Regression analysis attempts to determine the strength of the relationship between one dependent variable (usually denoted by Y) and a series of other changing variables (known as independent variables). For example, you might be evaluating how agents' language skill levels (independent, changing) affect their ability to achieve first contact resolution (the dependent variable).
How different Model types are evaluated for quality:
- For classification Models, which are evaluated using the area under the curve (AUC) method, you can analyze their effectiveness using a Receiver Operating Characteristic (ROC) Curve.
- Regression Models, such as those shown in the graphic, are evaluated by the following methods:
- RMSE (Root-mean-square deviation)
- MSE (Mean squared error)
- MAE (Mean absolute error)
- Agent % - This agent coverage metric indicates how many agent Models were built, as a function of the total agents available. It evaluates how much coverage a Model has. The agent coverage metric is available for Hybrid and Disjoint Model types.
- It is possible to 0% coverage. This indicates that none of the agents from the Agent Profile profile have records in the Dataset which was used to train the Model.
- Local Models - Displays the number of Local Models generated for agents in the Dataset on which the Predictor is built. Local Models are built only for Models that have the HYBRID or DISJOINT type. For GLOBAL Models this metric is always 0. If a Model is new (that is, inactive and untrained) the metric value is -1 until the Model is trained. Once trained, the metric shows the actual number of Local Models. Keep in mind the following requirement:
- For a numeric target metric, you must have at least one record per agent to build a Local Model.
- For a Boolean target metric, you must have at least one record per class (that is, one true/1 and one false/0) to build a Local Model.
Procedure: ROC Model Quality analysis
You can analyze the effectiveness of classification Models, which are evaluated by the area-under-curve (AUC) method, with a Receiver Operating Characteristic (ROC) Curve. This type of report is accessible only for classifier Predictors (Predictors that use a Boolean metric). Every trained Model for such a Predictor has an active AUC link in its row in the Predictors list.
- To open the Model Quality graph, click the AUC link in the Quality column.
- As you can see, the resulting diagram shows the ROC curve balancing the True Positive Rate and False Positive Rate. These terms, and some useful associated ones, are defined below:
- True Positive (TP) - The number items that met the specified condition, and were predicted to meet the condition. In this case, a TP would be an interaction that met the designated CSAT level (value =true) and was predicted correctly to do so.
- False Positive (FP) - The number of items that did not meet a condition, but were predicted to meet the condition. In this case, it would be interactions that were predicted to result in the specified CSAT level but did not.
- False Negative (FN) - The number of items that did meet a condition, but were predicted not to meet the condition. In this case, it would be interactions that were predicted to result in an unsatisfactory CSAT level but did not.
- Positive Population (PP) - The total number of interactions with a satisfactory CSAT (value = true).
- Negative Population (NP) - The total number of interactions with an unsatisfactory CSAT (value = false).
- The diagram shows a curve outlining the items that were true positives—correct, positive predictions-—that occurred at a rate better than guesswork (the black line). The analysis looks at Model sensitivity versus specificity.
- Sensitivity = TP/(TP + FN) = TP/PP - The ability of the test to detect the desired result.
- Specificity = TN/(TN + FP) = TN / NP - The ability of the test to correctly rule out the condition where it doesn't occur.
The ROC curve is a way to see the tradeoff between sensitivity and (1 - specificity) for different thresholds of probability for classification. This is the preferred way to view Model Quality for classifiers.
Activating a Model
Procedure: Activate your Model
After your Model is trained, activate it to use it for routing.
- Select the check box in the table row for your Model.
- Click the Play button.
- To deactivate a Model, click the Pause button.
Procedure: Activating multiple Models at once
You can test Model performance against other Models by activating multiple Models at once. You can then choose how much traffic is scored using each Model. Models are selected at random to score interactions, with the number scored by each Model dependent on how you weight the Model using the Mix parameter.
- In the Mix column, enter the desired numbers for each Model or set them using the up and down arrows located next to the number in the table cell.
- The numbers indicate the relative numbers of interactions that are scored using each Model. If you have a 1 in each column, the interactions are equally divided among the active Models. If you set 1 for Model A and 2 for Model B, Model B is used to route two interactions for every one using Model A.
- For example, you might have three Models to which you have assigned the weights of 1, 8, and 4 in the Mix field. Think of these weights as creating three buckets of different sizes, with a total value equal to the three weight values added together:
- 0 |_|______|___| 13
- On each incoming interaction, GPR chooses an entirely random number between 0 and the sum of all the weights (here this is 13). Depending on which bucket or range the chosen number falls in, the interaction is scored using the associated Model. In this example, Models are selected as follows:
- If the number is between 0 - 1 (the first bucket), the interaction is scored using the first Model
- If the number is between 2 - 9 (the second bucket), the interaction is scored using the second Model
- If the number is between 10 - 13 (the third bucket), the interaction is scored using the third Model
- Given that the numbers are selected randomly, the number of interactions scored using each Model end up proportional over time to the weights you selected in the Mix field.
Procedure: Editing Models
You can only edit a Model that has not yet been activated.
- If a Model has been trained and edited after training, the Model needs to be retrained.
Once a Model has been activated, it can not be changed or edited again. Even if you stop (deactivate) it, the Model remains locked.
To edit an activated Model, copy it. Only an active, trained Model can be copied. When you make a copy, you are creating a new Model that has the same name as the original Model with the added suffix <copy><version number appended>.
- Select the check box for the Model to be copied in the list of Models.
- The Copy button appears.
- Click Copy.
- The copy appears in the list of Models.
- Click the name of the copy and then follow the steps given above to edit, train, and activate the copy.
This page was last modified on April 24, 2019, at 12:29.