Using Split Testing to Compare Business Rule Outcomes
Overview
What problem does A/B/C Split Testing solve?
If your organization has a complex set of rules to define business decision-making, and you want to be able to compare alternative outcomes and scenarios, then rather than just changing a condition and hoping for the best you probably want to phase changes in slowly in order to measure the impact over time.
A/B/C Split Testing (hereafter, Split Testing) allows you to compare the business outcomes of alternative rule scenarios before rolling out significant changes to the way you make your business decisions. With Split Testing you can make, test and review changes incrementally to test their effects before committing to a particular change set.
For example, you might want to test several subtle changes in the way applications for credit cards are treated, in order to identify which has the best business outcome. You could split test for one income level against another, or one age range against another, or between different customer segments, or indeed any conditions in your rule package.
Flexible Approaches to Split Testing
Split Testing offers several approaches. For example, you can choose to:
- Unconditionally force either path A or B or C.
- Tie Split Testing to certain conditions (whether the customer is from a particular city, of a certain age, not a Gold customer, and so on).
- Weight the paths for A, B and C by percentage. This allows you to take a more statistical approach and use larger test data sets.
- Tie Split Testing to a business calendar (for example only do split testing on Fridays after 5 PM or Saturday morning 9 AM - 12 PM).
Split Testing Template
The GRSSplitTest_template.xml template provides some basic Facts (Conditions and Actions) for the Split Testing feature. To implement the feature, GRAT users must import this template (from the Samples folder) and attach it to all rules packages for which the Split Test functionality is required.
Split Test Column in Rule Packages
A Split Test column displays at the rule package level.
When the Split Test template has been imported into GRAT, this drop-down field will show the values imported from the template (A, B and C are the default values in the shipped template). If the template has not been imported, only the wildcard (*) symbol will be available.
Duplicate (copy and paste) the rule change that you wish to test. On the duplicated rule, make your logic change (for example, adjust the income requirement needed to offer a credit card).
Mark the original rule as “A” and the new rule as “B” by selecting from the “Split Test” column.
Once “A”, or “B” is selected, the rows color-code for easier viewing. The wildcard symbol (*) means that the Split Test condition is ignored, so this rule will always fire.
The rules marked as “A” or “B” (or “C”) will fire according to the logic in the Split Test Configuration. For example, you can weight the distribution such that “A” fires 75% of the time and “B” fires 25% of the time, etc. See the next section on Split Test Configuration for more options.
Split Test Configuration Node
A Split Test Configuration node appears in the left navigation tree. You can use this node to create (and display existing) Split Test rules that will apply at the rule package level. In rules created under this node, the Split Test column is hidden to avoid confusion.
What Can I Do with Split Test Rules?
Unconditionally force either path A or B or C
In rules created under this node, the splitTestValue can be selected as a Parameter for the rule. You can use this simple rule to force the Split Test path to any of the values in the template (the default shipped values in the template are A, B and C, but you can change these in the template if required).
Tie Split Testing to a Condition
To add more flexibility and precision to your use of Split Test rules, you can add to your rule any other condition available from the template(s) attached to the rule package. The example below has lower and upper age limits:
Weight the paths for A, B and C by percentage
You could add a condition to perform Weighted Distribution Split Testing. In this example the Split Test rule distributes the incoming rule evaluation requests across the A, B and C paths in the percentage proportions 25/25/50. This should look something like this:
Tie Split Testing to a Business Calendar
You can also tie Split Testing to a Business Calendar. For example, you might prefer to keep Split Testing outside normal business hours, or maybe run Split Testing in response to specific customer offers or other commercial events. To do this you can either use an existing Business Calendar or create a new one for your specific purpose. You then add this calendar to the rule to which you want to apply Split Testing.
So, when the parameters set for the Business Calendar apply, the weighted Split Testing begins and ends automatically.
Using Test Scenarios to Check Your Split Test Logic
GRAT's Test Scenario feature allows you to run pre-deployment checks on the logic of any rule package that you want to deploy. Because A/B/C Split Testing enables an additional level of logic to be built into a rule package, it's advisable to run more complex Split Test logic through a Test Scenario before deploying it to a production rules engine.
Example
1. You have a simple Call Treatment rule which states that only Gold customers will be offered a callback.
2. You want to test what would happen if you started offering callback to Gold and Silver customers. You decide you want to test this change by integrating it in for only 50% of the calls. So, you make the standard rule split test A:
3. The next step is to copy and paste the rule. Make the second rule split test B, then change the rule logic to allow Gold or Silver:
4. Next, create a simple Split Test rule, which will determine when to use rule A and when to use rule B. In this use case, you want this distributed 50% A and 50% B. So, select the action Split weight distribution 50% A, 50% B and 0% C. This action is provided in the Split Test example template.
5. Now, create a test scenario to test the changes. In this case, if the customer is Gold, you expect to always offer a callback. If the customer is Silver, you expect to offer a callback 50% of the time:
6. If you now run the test scenario multiple times, you will see that 50% of the time you get a green light on the Silver (indicating that you offered a callback) and 50% of the time you get a red light (indicating that you did not offer a callback):
First run
Second run
7. After testing that the A/B/C split logic is correct, you can deploy your rule package. At this point, the A/B/C split logic will run as you have it configured during normal rule evaluation. To adjust the A/B/C configuration (for example, make the A path 25% instead of 50% and the B path 75% instead of 25% and so on), you must make the changes in GRAT and then redeploy the rule package.
Allowing an Application to Override Split Test Configuration Rules
In addition to setting the A/B/C path in the Split Test Configuration section, it is possible for the invoking application (GVP, IVR, ORS and so on) to set the value. This can either be used as a default, or can be used to override the logic in the Split Test Configuration.
The application can pass in the split test value in the _GRS_Environment fact (field name: splitTestPath).
If you want the invoking application to be able to override the normal logic in Split Test Configuration, then you can simply add the Split Test Path is not set condition to your rule.
This condition will check whether the value has already been passed in by the invoking application. If it has been passed in, then the Split Test logic will be bypassed. If not, then the normal Split Test rule logic will apply.