Skip to main content

Writing logs to S3 with AWS Lambda

Modelbit logs can be sent via POST request to a server of your choice using the logs webhook integration.

By building a simple AWS Lambda function, we can route those logs to a storage system of our choice. In this tutorial, we'll write them to a S3 bucket we create.

Create your S3 bucket​

You can use any S3 bucket you own, but for simplicity and to avoid conflicts with other services, we'll create a new bucket for this tutorial. The bucket does not need any custom configuration or properties. The default AWS bucket configuration will do nicely.

Create a AWS Lambda function with function URL​

We'll need an AWS Lambda function that receives the logs and writes them to S3. The simplest way to do this is to use AWS's Lamdba function URLs. This lets us quickly create a REST-receiving AWS Lambda function without messing around with AWS API Gateways and other unnecessary complexity.

Start by creating a new function with a new function URL.

Go to the AWS Lambda functions page in the AWS console and click "Create a function". Select "Author from scratch," give your function a name, and select a Python runtime.

Before clicking Create function, expand "Additional configurations." Click "Enable" under "Function URL" and give it the following settings:

  • Auth type: None. Modelbit supports sending a secret parameter in order to implement your own authorization. We won't be using IAM to authenticate requests.
  • Check "Configure cross-origin resource sharing". Modelbit will be sending requests from its own URLs which will need to be accepted.

Here is the "Additional configurations" screen of a working logs-receiving Lambda function:

Once those options are selected, click "Create function"!

Give your Lambda function permissions on your S3 bucket​

Once the function is created, click the "Configuration" tab and then click "Permissions" in the sidebar. Lambda will show you the IAM role it created for your Lambda function. Click the role name to take you to AWS IAM.

Once there, look for the policy under "Permissions policies", click the "+" icon to expand it, and then click "Edit". We're going to add a permission to this policy.

Add this JSON to your IAM policy to let your Lambda function write files to your S3 bucket:

{
  "Effect": "Allow",
  "Action": "s3:PutObject",
  "Resource": ["arn:aws:s3:::<your bucket name>/*"]
}

Here's what the whole policy should look like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "logs:CreateLogGroup",
      "Resource": "arn:aws:logs:<region>:<account id>:*"
    },
    {
      "Effect": "Allow",
      "Action": ["logs:CreateLogStream", "logs:PutLogEvents"],
      "Resource": ["arn:aws:logs:<region>:<account id>:log-group:/aws/lambda/<lambda function name>:*"]
    },
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": ["arn:aws:s3:::<s3 bucket name>/*"]
    }
  ]
}

Save this policy and head back to your Lamdba function.

Write code in your Lambda function to write logs to S3​

From the "Code" tab of your Lambda function, you can write Python that receives logs and writes them to S3.

Here's example Python that makes a new S3 file for each incoming request and writes the JSON logs directly there:

import json
import time
import boto3

def lambda_handler(event, context):
    if "body" in event:
        logsStr = event["body"].encode("utf-8")

        bucketName = "<your bucket name>"
        fileName = f"{int(time.time())}.json"

        s3 = boto3.resource("s3")
        s3.Bucket(bucketName).put_object(Key=fileName, Body=logsStr)

    return {
        "statusCode": 200,
    }

You can click "Deploy" to deploy this function, "Test" to make test events to test it out, and customize to taste!

Add the Lambda function URL to Modelbit​

In the top "Function Overview" section of your Lambda function, in the bottom right, is the "Function URL." It'll look something like:

https://<long ugly string>.lambda-url.<region>.on.aws/

Click the handy copy icon next to it to copy it.

Head on over to Modelbit, click the Settings icon and then "Integrations." Click "Deployment Logs Webhook" and then paste your Lambda function URL into the "Webhook URL" field and click "Save Integration!"

Your Modelbit logs will now forward to your Lambda function, which will save them to S3. Happy logging!