Skip to main content

Request splitting

Request splitting allows you to run A/B tests among different versions of a deployment, or across different deployments, without the calling application needing to implement A/B testing logic.

Request splitting can also be used to roll out new versions of deployments with "canary" or "blue/green" release processes, where you control how much traffic the new version gets as it rolls out.

tip

To avoid errors, the deployments configured in each split should expect the same input format.

Creating a request splitting endpoint with the web app​

Click Endpoints in the header, then New Endpoint, and name your endpoint. Then click Splitting in the Endpoint type header, and configure your splits and split assignment.

Configuring request splits​

With request splitting you divide incoming requests among two or more "splits". Each split has a name, a deployment, and a weight.

The Name field is useful for analyzing the traffic later and will be logged with the request. Example names include control and experiment or variation_a, variation_b, and variation_c.

The Request handler specifies which deployment should receive requests forwarded from this endpoint, if the split is chosen. In some cases you may want to return a static value instead of calling a deployment. To do that, select Static value from the bottom of the list of deployments, and then supply a number or JSON object for the response.

The Weight field specifies the proportion of requests that should go to this split. For a 50%/50% ratio between two splits, choose equal weights (e.g. 1 and 1) for each split. For a 95%/5% split, use 95 and 5 instead.

Configuring split assignment​

By default request splitting uses the assignment setting By flipping a coin. This setting randomly assigns requests to splits, proportionally based on the weight of each splits.

To assign requests based on a parameter in the incoming request, choose the Based on a supplied key setting. The required Supplied key field should contain the name of the key in the dictionary of parameters sent to the deployment.

For example, if your deployment function expects a dictionary like { "user_id": 123, "category": "shopping", ... }, you could use user_id as the Subject key, and requests with the same user_id value will be randomly assigned to the same split.

If your deployment expects multiple arguments instead of a single dictionary, use an integer here instead. Use 0 to specify that the first argument to the deployment function should be used for split assignment, 1 for the 2nd argument, etc.

Alternatively, if you're also an Eppo customer, you can use Eppo for split assignment.

Creating a request splitting endpoint with git​

Endpoints are stored as .yaml files in the endpoints directory of your Modelbit git repo.

To create a new endpoint, create a new file in the endpoints directory. The name of the file will be the name of the endpoint. Use only lowercase letters, numbers, and underscores in the file name. This example endpoint is called pricing_optimizer and runs an 25%/75% A/B test between splits control and experiment.

endpoints/pricing_optimizer.yaml
backends:
  control:
    deployment:
      name: pricing_model_v1
      version: latest
    weight: 1
  experiment:
    deployment:
      name: pricing_model_v2
      version: latest
    weight: 3
schemaVersion: 1

Each group within the backends section represents a split, in this case control and experiment. Within each split is the deployment with a name and version, and a weight representing the proportion of traffic. The version field within deployment can be a numeric version or a version alias you've created.

Configuring split assignment​

By default, request are randomly assigned to a split, proportionally based on the weight of each split.

To assign requests based on a parameter in the incoming request, add an experimentConfig section and supply a subjectKey. The required subjectKey field should contain the name of the key in the dictionary of parameters sent to the deployment.

endpoints/pricing_optimizer.yaml
backends:
  control:
    deployment:
      name: pricing_model_v1
      version: latest
    weight: 1
  experiment:
    deployment:
      name: pricing_model_v2
      version: latest
    weight: 3
experimentConfig:
  experimentId: foo
  subjectKey: user_id
schemaVersion: 1

If your deployment expects multiple arguments instead of a single dictionary, use an integer here instead. Use 0 to specify that the first argument to the deployment function should be used for split assignment, 1 for the 2nd argument, etc.

You may optionally supply an experimentId, which acts as a random seed for split assignment when using subjectKey.

Returning static values​

To make the control arm of this experiment return a static value instead of calling a deployment, replace the control section with the following. This configuration returns {"score": 42 } as a static value:

control:
  value:
    score: 42
  weight: 1

Analyzing the results of A/B experiments​

Several additional fields are available in Modelbit logs for requests sent from an endpoint:

  • endpointName: The name of the endpoint that received the request that called the deployment.
  • experimentId: The experiment name, if configured, the endpoint used for request splitting.
  • backendName: The name of the split that received the request, calling the deployment.
  • results.subjectId is the value from subjectKey (from git) or Supplied key (from the web app).