CloudWatch JSON Logs: aws_request_id

Hello!

In a previous article, I showed how to make AWS lambda functions log JSON objects to CloudWatch (⬅️ start there if you’re new to JSON logs). The pattern in that post had a flaw: it didn’t pass the aws_request_id. I was writing small functions to glue together bits of deployment automation and I didn’t need it. It was easy to correlate logs just with timestamps. Not every case is that simple. Sometimes you need that ID. Fortunately, Paulie Pena IV gave me some tips on how to pass it through.

We can look up the aws_request_id in the lambda function’s context object. To add it to log events, the python-json-logger library supports custom fields. To add a field for all log events, we need to subclass jsonlogger.JsonFormatter. Then we can save the ID into the new field and resolve it in the format string. Here’s the code from before with that change:

import logging
from pythonjsonlogger import jsonlogger
 
class CustomJsonFormatter(jsonlogger.JsonFormatter):
    def __init__(self, *args, **kwargs):
        self.aws_request_id = kwargs.pop('aws_request_id')
        super().__init__(*args, **kwargs)
    def add_fields(self, log_record, record, message_dict):
        super().add_fields(log_record, record, message_dict)
        log_record['aws_request_id'] = self.aws_request_id
 
def setup_logging(log_level, aws_request_id):
    logger = logging.getLogger()
 
    # Testing showed lambda sets up one default handler. If there are more,
    # something has changed and we want to fail so an operator can investigate.
    assert len(logger.handlers) == 1
 
    logger.setLevel(log_level)
    json_handler = logging.StreamHandler()
    formatter = CustomJsonFormatter(
        fmt='%(aws_request_id)s %(asctime)s %(levelname)s %(name)s %(message)s',
        aws_request_id=aws_request_id
    )
    json_handler.setFormatter(formatter)
    logger.addHandler(json_handler)
    logger.removeHandler(logger.handlers[0])
 
def lambda_handler(event, context):
    setup_logging(logging.DEBUG, context.aws_request_id)
    logger = logging.getLogger()
    logger.info('Huzzah!')

Now our logs contain the aws_request_id:

WithRequestId

Hope that helps,

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

Lambda Gotcha: CloudWatch Logs Group Name

Hello!

Today’s post is a little “gotcha” that sometimes still gets me when I’m developing AWS lambda functions: if you want to stream the function’s logs to CloudWatch the log group’s name has to follow a specific convention.

Suppose I’m creating a lambda function with this CloudFormation snippet:

Function:
  Type: AWS::Lambda::Function
  Properties:
    Code:
      ZipFile: |
        # https://operatingops.com/2018/10/13/cloudformation-custom-resources-avoiding-the-two-hour-exception-timeout/
        import logging
        import cfnresponse
 
        def handler(event, context):
            logger = logging.getLogger()
            logger.setLevel(logging.INFO)
            try:
                if event['RequestType'] == 'Delete':
                    logger.info('Deleted!')
                    cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
                    return
 
                logger.info('It worked!')
                cfnresponse.send(event, context, cfnresponse.SUCCESS, {})
            except Exception:
                logger.exception('Signaling failure to CloudFormation.')
                cfnresponse.send(event, context, cfnresponse.FAILED, {})
    FunctionName: custom-resource
    Handler: index.handler
    Role: !GetAtt ExecutionRole.Arn
    Runtime: python3.7
    Timeout: 30

The key piece is this:

FunctionName: custom-resource

When AWS lambda sends logs to CloudWatch, it assumes the target log group has a name like this:

/aws/lambda/[function name]

This isn’t configurable. If your log group’s name doesn’t follow this convention you won’t get logs from your lambda function.

So, in our case, we need a log group called /aws/lambda/custom-resource. In CloudFormation, we could create it like this:

Logs:
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: /aws/lambda/custom-resource
    RetentionInDays: 30

The IAM role attached to your function of course still needs permissions to send logs, and there’s another gotcha there that can lead to orphaned log groups.

Hope this helps!

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles:

CloudWatch Logs: Preventing Orphaned Log Groups

Hello!

When you need to publish logs to CloudWatch (e.g. from a lambda function), you need an IAM role with access to CloudWatch. It’s tempting to use a simple policy like the one in the AWS docs. You might write a CloudFormation template like this:

# Don't use this!
 
AWSTemplateFormatVersion: '2010-09-09'
 
Resources:
  DemoRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: '/'
      Policies:
      - PolicyName: lambda-logs
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
          - Effect: Allow
            Action:
            - logs:CreateLogGroup
            - logs:CreateLogStream
            - logs:DescribeLogStreams
            - logs:PutLogEvents
            Resource: arn:aws:logs:*:*:*
 
  DemoFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
          def handler(event, context):
              print('Demo!')
      FunctionName: demo-function
      Handler: index.handler
      Role: !GetAtt DemoRole.Arn
      Runtime: python3.7

Obviously, the role is too permissive: arn:aws:logs:*:*:*

But, there’s another problem: it grants logs:CreateLogGroup.

Here’s what happens:

  1. Launch a stack from this template
  2. Run demo-function
  3. Because we granted it permission, demo-function automatically creates /aws/lambda/demo-function log group in CloudWatch Logs
  4. Delete the stack
  5. CloudFormation doesn’t delete the /aws/lambda/demo-function log group

CloudFormation doesn’t know about the function’s log group because it didn’t create that group, so it doesn’t know anything needs to be deleted. Unless an operator deletes it manually, it’ll live in the account forever.

It seems like we can fix that by having CloudFormation create the log group:

DemoLogGroup:
  Type: AWS::Logs::LogGroup
  Properties:
    LogGroupName: /aws/lambda/demo-function
    RetentionInDays: 30

But, if the function still has logs:CreateLogGroup I’ve seen race conditions where the stack deletes the group before the lambda function and the function recreates that group before it gets deleted.

Plus, there aren’t any errors if you forget to define the group in CF. The stack launches. The lambda function runs. We even get logs, they’ll just be orphaned if we ever delete the stack.

That’s why it’s a problem to grant logs:CreateLogGroup. It allows lambda (or EC2 or whatever else is logging) to log into unmanaged groups.

All resources in AWS should be managed by CloudFormation (or terraform or whatever resource manager you use). Including log groups. So, you should never grant logs:CreateLogGroup except to your resource manager. Nothing else should need that permission.

And that’s the other reason: lambda doesn’t need logs:CreateLogGroup because it should be logging to groups that already exist. You shouldn’t grant permissions that aren’t needed.

Here’s the best practice: always manage your CloudWatch Logs groups and never grant permission to create those groups except to your resource manager.

Happy automating!

Adam

Need more than just this article? We’re available to consult.

You might also want to check out these related articles: