Request splitting
Request splitting allows you to run A/B tests among different versions of a deployment, or across different deployments, without the calling application needing to implement A/B testing logic.
Request splitting can also be used to roll out new versions of deployments with "canary" or "blue/green" release processes, where you control how much traffic the new version gets as it rolls out.
To avoid errors, the deployments configured in each split should expect the same input format.
Creating a request splitting endpoint with the web app
Click Endpoints
in the header, then New Endpoint
, and name your endpoint. Then click Splitting
in the Endpoint type
header, and configure your splits and split assignment.
Configuring request splits
With request splitting you divide incoming requests among two or more "splits". Each split has a name, a deployment, and a weight.
The Name
field is useful for analyzing the traffic later and will be logged with the request. Example names include control
and experiment
or variation_a
, variation_b
, and variation_c
.
The Request handler
specifies which deployment should receive requests forwarded from this endpoint, if the split is chosen. In some cases you may want to return a static value instead of calling a deployment. To do that, select Static value
from the bottom of the list of deployments, and then supply a number or JSON
object for the response.
The Weight
field specifies the proportion of requests that should go to this split. For a 50%/50% ratio between two splits, choose equal weights (e.g. 1
and 1
) for each split. For a 95%/5% split, use 95
and 5
instead.
Configuring split assignment
By default request splitting uses the assignment setting By flipping a coin
. This setting randomly assigns requests to splits, proportionally based on the weight of each splits.
To assign requests based on a parameter in the incoming request, choose the Based on a supplied key
setting. The required Supplied key
field should contain the name of the key in the dictionary of parameters sent to the deployment.
For example, if your deployment function expects a dictionary like { "user_id": 123, "category": "shopping", ... }
, you could use user_id
as the Subject key
, and requests with the same user_id
value will be randomly assigned to the same split.
If your deployment expects multiple arguments instead of a single dictionary, use an integer here instead. Use 0 to specify that the first argument to the deployment function should be used for split assignment, 1 for the 2nd argument, etc.
Alternatively, if you're also an Eppo customer, you can use Eppo for split assignment.
Creating a request splitting endpoint with git
Endpoints are stored as .yaml
files in the endpoints
directory of your Modelbit git repo.
To create a new endpoint, create a new file in the endpoints
directory. The name of the file will be the name of the endpoint. Use only lowercase letters, numbers, and underscores in the file name. This example endpoint is called pricing_optimizer
and runs an 25%/75% A/B test between splits control
and experiment
.
backends:
control:
deployment:
branch: main
name: pricing_model_v1
version: latest
weight: 1
experiment:
deployment:
branch: main
name: pricing_model_v2
version: latest
weight: 3
schemaVersion: 1
Each group within the backends
section represents a split, in this case control
and experiment
. Within each split is the deployment
with a name
and version
, and a weight
representing the proportion of traffic. The version
field within deployment
can be a numeric version or a version alias you've created.
Configuring split assignment
By default, request are randomly assigned to a split, proportionally based on the weight of each split.
To assign requests based on a parameter in the incoming request, add an experimentConfig
section and supply a subjectKey
. The required subjectKey
field should contain the name of the key in the dictionary of parameters sent to the deployment.
backends:
control:
deployment:
branch: main
name: pricing_model_v1
version: latest
weight: 1
experiment:
deployment:
branch: main
name: pricing_model_v2
version: latest
weight: 3
experimentConfig:
experimentId: foo
subjectKey: user_id
schemaVersion: 1
If your deployment expects multiple arguments instead of a single dictionary, use an integer here instead. Use 0 to specify that the first argument to the deployment function should be used for split assignment, 1 for the 2nd argument, etc.
You may optionally supply an experimentId
, which acts as a random seed for split assignment when using subjectKey
.
Returning static values
To make the control
arm of this experiment return a static value instead of calling a deployment, replace the control
section with the following. This configuration returns {"score": 42 }
as a static value:
control:
value:
score: 42
weight: 1
Analyzing the results of A/B experiments
Several additional fields are available in Modelbit logs for requests sent from an endpoint:
endpointName
: The name of the endpoint that received the request that called the deployment.experimentId
: The experiment name, if configured, the endpoint used for request splitting.backendName
: The name of the split that received the request, calling the deployment.results.subjectId
is the value fromsubjectKey
(from git) orSupplied key
(from the web app).