Copying your API logs into Snowflake with Snowpipe
First, configure your Warehouse to access the API logs stage. Your warehouse-specific sample code can be found in
the "Settings" tab in Modelbit. Click your warehouse name in the left menu, then complete the steps
under Integrate Storage for Access to Logs
.
Then in Settings click Integrations, and follow the instructions under Snowflake Snowpipe
. Snowpipe will only collect
new API logs, so run a deployment at least once to generate new log entries for Snowpipe to ingest.
Here are some sample queries for viewing your logs after they've been ingested with Snowpipe:
-- see all raw log data
select * from <LOGS_SCHEMA>.modelbit_logs_raw;
-- expand json blobs to rows
with api_stats as (
select
to_timestamp((j.value:batchStartTs/1000)::int) as batch_start,
j.value:batchDurationMs::int as batch_duration_ms,
j.value:deploymentName as deployment_name,
j.value:deploymentVersion as deployment_version,
strip_null_value(j.value:error) as error,
j.value:resultCount::int as result_count,
VALUE as complete_log_line
from <LOGS_SCHEMA>.modelbit_logs_raw, lateral flatten(input => "DATA") as j
where error is null
)
select * from api_stats
order by 1 desc
limit 100;
Available Log Fields
The following fields are available in the JSON-formatted log data:
deploymentName
: The name of the deployment (string
).deploymentVersion
: The numeric version of the deployment (string
).alias
: The version alias (e.g.v1.2
) orlatest
if the deployment was called using an alias instead of a numeric version (string|null
).branch
: The git branch of the deployment (string
).batchStartTs
: The timestamp, in milliseconds, when this request started (int
).batchEndTs
: The timestamp, in milliseconds, when this request completed (int
).batchDurationMs
: The latency of the request (e.g. time experienced by callers of the deployment). It's the difference betweenbatchStartTs
andbatchEndTs
(int
).deploymentRuntimesMs
: Modelbit may split the list of inputs to deployments over many concurrently-executing instances of the deployment to improve performance. This field shows the individual runtime durations of each instance used in this batch (list[int]
).totalDeploymentRuntimeMs
: The total time spent running this batch. It's the sum ofbatchDurationMs
anddeploymentRuntimesMs
(int
).resultCount
: The number of results returned (int
).results
: A list of objects describing the inputs and outputs of this model call (list[object]
):id
: The input ID supplied during request call. If no ID was supplied, it's the row's index in the result list.inputs
: A list of argument sent to the deployment's function (list[any]
).output
: The value returned from the deployment's function (any
).stdout
: The text from anyprint(...)
during execution. (null|object
).
source
: The origin of the request (rest|snowflake|redshift|tests
).error
: The error message if one was triggered by the deployment's function (string|null
).workspaceId
: Your Modelbit workspace ID (string
).batchId
: The ID of this particular batch (string
).apiKeyId
: The ID of the key used to call the deployment, if API keys were used (string|null
).
Helpful Status Queries
To see the current status of the Snowpipe:
select SYSTEM$PIPE_STATUS( '<LOGS_SCHEMA>.modelbit_logs_pipe' );
To see the history of the Snowpipe's executions:
select * from table(
information_schema.pipe_usage_history(
pipe_name=>'<LOGS_SCHEMA>.modelbit_logs_pipe'));