Skip to main content

Best practices for evaluating new deployment versions

You can deploy a new version of your model to production and test its performance against old versions of your model in production.

Developing and deploying the first version​

It is recommended to develop the first version of your model on your own development git branch. Once you are ready to use your model in production, merge your branch to main using your favorite git tool, a GitHub Pull Request, or a GitLab Merge Request.

You will now be able to call your model using a REST endpoint like:

https://my-company.app.modelbit.com/v1/my_model/1

And if you have a Snowflake or Redshift warehouse connected, with a SQL function like:

select my_schema.ext_my_model_1(...);
tip

While Modelbit supports a latest pointer to automatically call the latest version of a model, that is not best practice when you plan to evaluate new versions of a model with an A/B test. Use /my_model/1 instead.

Developing the next version and testing before production​

Once your first model is running in production, you may want to develop a new version and test it locally or in a staging environment. Return to your branch to do further development. If you deleted your branch as part of the Pull or Merge Request, create a new branch for further development.

You can deploy your model while working on the branch. This will create a deployment only on that branch, with a URL like:

https://my-company.app.modelbit.com/v1/model_predict/my_development_branch

And a SQL function like:

select my_schema.ext_my_model_my_development_branch(...);

These endpoints are ideal for testing the updated model's behavior as compared to the version in production.

Testing in production​

You can test a new model version in production, e.g. with an A/B test comparing it to the previous version.

To do this, go ahead and merge the development branch with the new model into main, using GitHub, GitLab or your favorite git tool. This will deploy a new version of your model in production. If the previously deployed version was 1, this version will be 2. It will have a URL like:

https://my-company.app.modelbit.com/v1/my_model/2

And a SQL function like:

select my_schema.ext_my_model_2(...);

This does not impact your v1 running model. If you are calling the endpoint that ends in my_model/1, that endpoint still accesses the v1 model. Now you can change your application's behavior to send some calls to the endpoints ending in my_model/2.

Comparing model outputs in production​

With some calls going to v1 and some calls going to v2 in production, you can compare the outputs. Navigating to version 1 in the left hand navigation and selecting the "Logs" tab will show you the runtime logs of that version:

Choosing version 2 in the version picker will show you the logs from v2 of your deployment:

These logs are also separated in Modelbit's logs integration. In each JSON log line, deploymentVersion will be 1 for the first production version, 2 for the second production version, and the name of your branch for any versions deployed from branches for testing.

When to make a brand new deployment​

In general, making new deployments to test new versions of the same model is not necessary or recommended. The exceptions are:

  • If you plan to run both versions in production indefinitely, especially if you plan to iterate both versions independently. In this case the two versions can be thought of as distinct models over time.
  • If you are migrating an already-existing model into Modelbit, and already have different code for both model versions. In this case, if you find it more convenient to deploy this way, consider having a my_model_legacy deployment with the older version, and a my_model deployment with the newer version. Once you have directed all API traffic to the newer version, you can delete the older version and use the recommended workflow for future updates.