Skip to main content

Request splitting & A/B testing

Request splitting allows you to run A/B tests among different versions of a deployment without the calling application needing to implement A/B testing logic.

Request splitting can also be used to roll out new versions of deployments with "canary" or "blue/green" release processes, where you control how much traffic the new version gets as it rolls out.

tip

To avoid errors, the deployments configured in each split should expect the same input format.

Creating a request splitting endpoint with the web app​

Click Endpoints in the header, then New Endpoint, and name your endpoint. Then click Splitting in the Endpoint type header, and configure your splits and split assignment.

Configuring request splits​

With request splitting you divide incoming requests among two or more "splits". Each split has a name, a deployment, and a weight.

The Name field is useful for analyzing the traffic later and will be logged with the request. Example names include control and experiment or variation_a, variation_b, and variation_c.

The Request handler specifies which deployment should receive requests forwarded from this endpoint, if the split is chosen.

The Weight field specifies the proportion of requests that should go to this split. For a 50%/50% ratio between two splits, choose equal weights (e.g. 1 and 1) for each split. For a 95%/5% split, use 95 and 5 instead.

Creating a request splitting endpoint with git​

Endpoints are stored as .yaml files in the endpoints directory of your Modelbit git repo.

To create a new endpoint, create a new file in the endpoints directory. The name of the file will be the name of the endpoint. Use only lowercase letters, numbers, and underscores in the file name. This example endpoint is called pricing_optimizer and runs an 25%/75% A/B test between splits control and experiment.

endpoints/pricing_optimizer.yaml
backends:
  control:
    deployment:
      branch: main
      name: pricing_model_v1
      version: latest
    weight: 1
  experiment:
    deployment:
      branch: main
      name: pricing_model_v2
      version: latest
    weight: 3
schemaVersion: 1

Each group within the backends section represents a split, in this case control and experiment. Within each split is the deployment with a name and version, and a weight representing the proportion of traffic. The version field within deployment can be a numeric version or a version alias you've created.

Analyzing the results of A/B experiments​

Several additional fields are available in Modelbit logs for requests sent from an endpoint:

  • endpointName: The name of the endpoint that received the request that called the deployment.
  • backendName: The name of the split that received the request, calling the deployment.