Jump to: navigation, search

Configuring, Training, and Testing Models

A model is built on a predictor and includes the same target metric. Each model has a subset of the agent and customer features present in the dataset. The Feature Analysis report helps you to identify the features with the strongest impact on the target metric. You can create multiple models for the same predictor, each with a different set of features selected.

  • You can compare how well models work to create the most effective ones.
  • You can configure models that are best-suited to specific circumstances and control which model is used by activating and deactivating them. This enables you to respond promptly to changes such as weekday vs weekend volume or the anticipated increase in certain types of interactions after a big marketing push.

When you create a predictor, a full feature set model is created automatically, including all the agent and customer features populated from the predictor. A predictor can have a number of models associated with it.

Important

The full feature set model requires a complete set of local models, one for each agent. If it's not possible to train a local model for a specific agent, GPR generates the following log message, indicating that the local model model has been skipped for a specific agent, and does not create the full feature set model: WARNING <BOTTLE> models.py:420 RETRAIN: Skipping training for individual model <agent_id> since no data is available for classes.

For example, you might have a Boolean metric, such as FCR. If, for a specific agent, all entries from the training set had only examples of one Boolean class (that is, either all were resolved, or none were resolved), GPR cannot create a local model for that agent.

  • You can configure a routing strategy to use a specific predictor, then edit the models and change which are active. In this way, routing can be adjusted and optimized on the fly, without requiring you to edit the strategy. For instructions on this specific functionality, see Activating multiple models at once, below.
  • The list of models includes the Quality column, which provides analysis reports on model quality and agent coverage.
  • You can export an existing model as a JSON file and then import it, rather than creating it over from the beginning. For model import/export information, see Exporting a model.

To open the configuration menu, click the Settings gear icon, located on the right side of the top menu bar: GPMsettingsGear.png.

Open the models interface

1

To access your models:

  1. Select Predictor from the left-hand navigation bar and then click the name of a predictor in your list.
  2. Click Models.

Add a new model

1

To create a new model, click Add Predictor Model.

Configure a model

1

You can define new model using the following steps:

  1. Edit the fields as shown in the graphic.
    Important
    You can only select from the Agent Features and Customer Features that are included in the predictor.
  2. To choose Agent Features and Customer Features, click in the appropriate text box and select the desired features from the drop-down list.
    • Agent Features are items that refer to the agent. All agent-related fields in your selected dataset appear in the Agent Features field.
    • Customer Features are items that refer to the customer or that are available in interaction user data. They refer to aspects of the environment, broadly speaking, in which the interaction occurs.
  3. To remove a feature, click the X in the box containing the feature name.
  4. Choose one of the model types, GLOBAL, DISJOINT, or HYBRID.

Global Model

  • A single model is built to predict agent scores.
  • Provides generalizations that are probably true across the whole pool of agents. For example, agents that are part of this group might have a lower transfer rate; agents with longer tenure generally have higher resolution performance; and so on.
  • Select this model type only if there are action features that correlate with the target metric.
  • In the absence of Agent Features, the Global model type produces the same score for every agent, which makes it useless for ranking.

Disjoint Model

  • A model, also called a local model, is built for each agent.
  • Captures agent-specific idiosyncrasies with respect to performance.
  • With this option, a Global model is also built to produce scores for agents for whom individual models cannot be created. For example, some agents might not have enough data to create an individual model.To build a model for a numeric metric, an agent must have at least one interaction; to build a model for a Boolean metric, an agent must have at least one interaction for each of the possible Boolean values.

Hybrid Model

  • Combination of Global and Disjoint model, which uses an average of their scores.
  • Can both provide generalization and capture agent-specific nuances in performance.
  • Select this model only if there are Agent Features that correlate with the target metric; that is, the metric (first-contact resolution, net promoter score, average handle time, and so on) for which the associated predictor is built.

Set the training and testing percentages

1

You can configure how much of your dataset is used to train your model and how much is used to test how well it works. This split is time-based. The most recent interactions are allocated to the test section of the dataset. For example, if you use 80% of the data to train your model and 20% to test it, the most recent 20% of the dataset records are used for testing.

To set this value:

  1. Use your mouse to slide the indicator to the desired point on the Train/Test bar.

Train your new model

1

After you create your model, you must train it on your data. When a trained model has not been yet activated, you can modify it. For example, you might add new Agent or Customer Feature, change the train/test split, and so on. After those modifications, you must retrain your model. When you train or retrain a model, the integer in the Version column of the list of models is incremented.

If you change the date range for the predictor data, then purge and regenerate your dataset, an already-trained model does not need to be retrained. It uses the previously configured date range. But all models created after the data purge and regeneration use the new date range.

  1. Select the check box in the table row for your model.
  2. Click Train.

The Training job can take a fairly long time, depending on the size of your dataset. Click the Jobs tab to monitor job progress.

View model quality and agent coverage reports

1

After you train your model, the Quality column shows values for various methods of evaluating how well the model works. The evaluation methods are selected automatically depending on the type of model.

  • Classification analysis buckets observations into predetermined categories, based on data already used for training. In this case, you already know that the data can be divided into meaningful categories, into which your new data can be placed. For example, you might record interaction results showing whether a customer's issue was resolved or not; whether the desired AHT was met or not; whether the final NPS was above a certain value or not; and so on.
  • Regression analysis attempts to determine the strength of the relationship between one dependent variable (usually denoted by Y) and a series of other changing variables (known as independent variables). For example, you might be evaluating how agents' language skill levels (independent, changing) affect their ability to achieve first contact resolution (the dependent variable).

How different model types are evaluated for quality:

  • For classification models, which are evaluated using the area under the curve (AUC) method, you can analyze their effectiveness using a Receiver Operating Characteristic (ROC) Curve.
  • Regression models, such as those shown in the graphic, are evaluated by the following methods:
    • RMSE (Root-mean-square deviation)
    • MSE (Mean squared error)
    • FBQ
    • MAE (Mean absolute error)
    • R2S
  • Agent % - This agent coverage metric indicates how many agent models were built, as a function of the total agents available. It evaluates how much coverage a model has. The agent coverage metric is available for Hybrid and Disjoint model types.
  • Local models - Displays the number of local models generated for agents in the dataset on which the predictor is built. Local models are built only for models that have the HYBRID or DISJOINT type. For GLOBAL models this metric is always 0. If a model is a new (that is, inactive and untrained) the metric value is -1 until the model is trained. Once trained, the metric shows the actual number of local models. Keep in mind the following requirement:
    • For a numeric target metric, you must have at least one record per agent to build a local model.
    • For a Boolean target metric, you must have at least one record per class (that is, one true/1 and one false/0) to build a local model.

ROC model quality analysis

1

You can analyze the effectiveness of classification models, which are evaluated by the area-under-curve (AUC) method, with a Receiver Operating Characteristic (ROC) Curve.This type of report is accessible only for classifier predictors (predictors that use a boolean metric). Every trained model for such a predictor has an active AUC link in its row in the Predictors list.

  • To open the model quality graph, click the AUC link in the Quality column.

As you can see, the resulting diagram shows the ROC curve balancing the True Positive Rate and False Positive Rate. These terms, and some useful associated ones, are defined below:

  • True Positive (TP) - The number items that met the specified condition, and were predicted to meet the condition. In this case, a TP would be an interaction that met the designated CSAT level (value =true) and was predicted correctly to do so.
  • False Positive (FP) - The number of items that did not meet a condition, but were predicted to meet the condition. In this case, it would be interactions that were predicted to result in the specified CSAT level but did not.
  • False Negative (FN) - The number of items that did meet a condition, but were predicted not to meet the condition. In this case, it would be interactions that were predicted to result in an unsatisfactory CSAT level but did not.
  • Positive Population (PP) - The total number of interactions with a satisfactory CSAT (value = true).
  • Negative Population (NP) - The total number of interactions with an unsatisfactory CSAT (value = false).

The diagram shows a curve outlining the items that were true positives--correct, positive predictions--that occurred at a rate better than guesswork (the green line). The analysis looks at model sensitivity versus specificity.

  • Sensitivity = TP/(TP + FN) = TP/PP - The ability of the test to detect the desired result.
  • Specificity = TN/(TN + FP) = TN / NP - The ability of the test to correctly rule out the condition where it doesn't occur.

The ROC curve is a way to see the tradeoff between sensitivity and (1 - specificity) for different thresholds of probability for classification. This is the preferred way to view model quality for classifiers.

Activate your model

1

After your model is trained, activate it to use it for routing.

  1. Select the check box in the table row for your model.
  2. Click the Play button.
  3. To deactivate a model, click the Pause button.

Activating multiple models at once

1

You can test model performance against each other by activating multiple models at once. You can then choose how much traffic goes to each model.

  • In the Mix column, enter the desired numbers for each model or set them using the up and down arrows located next to the number in the table cell.
    The numbers indicate the relative numbers of interactions that go to each model. If you have a 1 in each column, half the interactions go to each model. If you set 1 for Model A and 2 for Model B, Model B will get two interactions for every one going to Model A.

Editing models

1

You can only edit a model that has not yet been activated.

  • If a model has been trained and edited after training, the model needs to be retrained.

Once a model has been activated, it can not be changed or edited again. Even if you stop (deactivate) it, the model remains locked.

To edit an activated model, copy it. Only an Active Trained model can be copied. When you make a copy, you are creating a new model that has the same name as the original model with the added suffix <copy><version number appended>.

  1. Select the check box for the model to be copied in the list of models.
    The Copy button appears.
  2. Click Copy.
    The copy appears in the list of models.
  3. Click the name of the copy and then follow the steps given above to edit, train, and activate the copy.

Exporting a model

1

Exporting a model enables you to move models from one environment to another. You can export a model from the the page where you configure it. You can choose to import the resulting JSON file when creating or updating a predictor.

For example, you might create, train, and test a model in a test environment, then use the export/import functionality to move the model into your production environment. The Export button becomes available when you drill-down to a specific model and open it for editing, as shown in the graphic.

  • You can only export a model that has been activated.

To import a model:

  • Open a predictor and click the Select Model JSON File button.

Feedback

Comment on this article:

blog comments powered by Disqus
This page was last modified on October 19, 2018, at 17:26.