Copying your API logs into Snowflake with Snowpipe

First, configure your Warehouse to access the API logs stage. Your warehouse-specific sample code can be found in the "Settings" tab in Modelbit. Click your warehouse name in the left menu, then complete the steps under Integrate Storage for Access to Logs.

Then in Settings click Integrations, and follow the instructions under Snowflake Snowpipe. Snowpipe will only collect new API logs, so run a deployment at least once to generate new log entries for Snowpipe to ingest.

Querying the logs

Check out the deployment logs page for the fields and format of log lines.

Here are some sample queries for viewing your logs after they've been ingested with Snowpipe:

-- see all raw log data
select * from <LOGS_SCHEMA>.modelbit_logs_raw;

-- expand json blobs to rows
with api_stats as (
   select
      to_timestamp((j.value:batchStartTs/1000)::int) as batch_start,
      j.value:batchDurationMs::int as batch_duration_ms,
      j.value:deploymentName as deployment_name,
      j.value:deploymentVersion as deployment_version,
      strip_null_value(j.value:error) as error,
      j.value:resultCount::int as result_count,
      VALUE as complete_log_line
   from <LOGS_SCHEMA>.modelbit_logs_raw, lateral flatten(input => "DATA") as j
   where error is null
)
select * from api_stats
order by 1 desc
limit 100;

Helpful status queries

To see the current status of the Snowpipe:

select SYSTEM$PIPE_STATUS( '<LOGS_SCHEMA>.modelbit_logs_pipe' );

To see the history of the Snowpipe's executions:

select * from table(
   information_schema.pipe_usage_history(
      pipe_name=>'<LOGS_SCHEMA>.modelbit_logs_pipe'));