Adding inference timeouts

Modelbit's REST API has a default timeout of 5 minutes. You can control that timeout with the timeout_seconds parameter.

For example, to apply a 60-second timeout to your online inference requests:

modelbit.get_inference

Use the timeout_seconds parameter of modelbit.get_inference:

import modelbit

modelbit.get_inference(..., data=..., timeout_seconds: 60)

requests

Add the timeout_seconds parameter as a sibling to your data:

import json, requests

requests.post("https://...", data=json.dumps({"data": ..., "timeout_seconds": 60})).json()

curl

Add the timeout_seconds parameter as a sibling to your data:

curl -s -XPOST "https://..." -d '{"data": ..., "timeout_seconds": 60}'

Inference requests that take longer than the timeout will be terminated and an error will be returned.

warning

Deployments that exceed the timeout are stopped and restarted. The next request after a timeout will be a cold start.

Longer timeouts for async requests

To use a timeout longer than 5 minutes you must also use the async REST API. The maximum timeout when using the async API is 15 minutes.