How to debug Lambda to ec2 - AWS policy?

Example lambda extract

This lambda listens to cloudwatch log and then triggers s3 sync to an ec2 instance id

const sendCommandParams: SendCommandCommandInput = {
    DocumentName: "AWS-RunShellScript",
    Parameters: {
      commands: [`aws s3 sync s3://${BUCKET_NAME} ${DOCKER_VOLUME_PATH} --delete --no-progress`],
    },
    InstanceIds: [ec2InstanceId],
  };
  const sendCommandCommand = new SendCommandCommand(sendCommandParams);
  const response = await ssmClient.send(sendCommandCommand);

Cloudwatch error

ERROR Error: AccessDeniedException: User: arn:aws:sts::$accountID:assumed-role/role-lambdaEC2okOK/lambdaEC2okOK is not authorized to perform: ssm:SendCommand on resource: arn:aws:ssm:$REGION::document/AWS-RunShellScript because no identity-based policy allows the ssm:SendCommand action

EC2 instance error

• ssh into ec2 instance - debug ssm agent logs

sudo tail -f /var/log/amazon/ssm/amazon-ssm-agent.log

024-11-17 07:59:50.4887 INFO [ssm-agent-worker] [MessageService] [MGSInteractor] SSM Connection channel status is set to ec2messages
2024-11-17 07:59:50.4887 ERROR [ssm-agent-worker] [MessageService] [MGSInteractor] Failed to get controlchannel token, error: CreateControlChannel failed with error: createControlChannel request failed: unexpected response from the service
User: arn:aws:sts::$accountID:assumed-role/role-ec2-instance/$instanceID is not authorized to perform: ssmmessages:CreateControlChannel on resource: arn:aws:ec2:$REGION:$accountID:instance/$instanceID because no identity-based policy allows the ssmmessages:CreateControlChannel action

Add Missing policy on ec2 policy

• smmessages:CreateControlChannel

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:UpdateInstanceInformation"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2messages:AcknowledgeMessage",
        "ec2messages:DeleteMessage",
        "ec2messages:FailMessage",
        "ec2messages:GetEndpoint",
        "ec2messages:GetMessages",
        "ec2messages:SendReply"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads",
        "s3:ListMultipartUploadParts",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}",
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    }
  ]
}

Update Lambda Policy

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/AWS-RunShellScript",
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/AWS-RunShellScript",
        "arn:aws:ssm:${REGION}:${accountID}:managed-instance/*",
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/AWS-RunShellScript"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:document/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:instance/*"
      ]
    }

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
         "*:*:*:${REGION}:${accountID}:*"
      ]
    }

Fail

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ssm:${REGION}:${accountID}:*/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "arn:aws:ec2:${REGION}:${accountID}:*/*"
      ]
    },

Success - Unfortunately can't restrict any more

{
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "*"
      ]
    }

Complete policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2messages:AcknowledgeMessage",
        "ec2messages:DeleteMessage",
        "ec2messages:FailMessage",
        "ec2messages:GetEndpoint",
        "ec2messages:GetMessages",
        "ec2messages:SendReply"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:SendCommand"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetDocument",
        "ssm:GetCommandInvocation",
        "ssm:ListAssociations",
        "ssm:ListInstanceAssociations",
        "ssm:DescribeInstanceInformation"
      ],
      "Resource": [
        "*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::${BUCKET_NAME}",
        "arn:aws:s3:::${BUCKET_NAME}/*"
      ]
    }
  ]
}

Gotchas

The odd bug - when the ec2 instance isn't ready - despite it passing system status check:

ec2 describe-instance-status

• SystemStatus.Status = "ok"

• InstanceStatus.Status= "ok"

  InvalidInstanceId: Instances [[i-044df8615ccf4805c]] not in a valid state for account $baccountID

• Inside Lambda allow an additional 500ms if error?
  Must separate Arns for SSM
  Some policies cannot be restricted on Resource - ie 

• ec2messages:*

• ssm:* except ssm:SendCommand

Notes:

Cloud watch logs don't always update with timestamps

• Go into last cloudwatch log stream and refresh

Lambda needs additional basic permission:

• arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole

Need both:*

• ec2 instance policy to allow ssm communication 
• Lambda policy policy to allow ssm communication

Ec2 instance security update to allow ssm communication

aws ec2 authorize-security-group-ingress \
    --group-id $securityGroupID \
    --protocol tcp \
    --port 443 \
    --cidr 0.0.0.0/0 \
    --query="SecurityGroupRules[].SecurityGroupRuleId"

Test lambda on local computer AND in situ, ie when triggered by cloudwatch log or other because it may work in one but not the other

If invoking lambda via cloudwatch log - then lambda needs additional permission:

aws lambda add-permission \
 --function-name $LAMBDA_FUNCTION_EC2_OKOK \
 --statement-id AllowCloudWatchLogsInvoke \
 --action "lambda:InvokeFunction" \
 --principal logs.amazonaws.com \
 --source-arn "arn:aws:logs:${REGION}:${accountID}:log-group:${project_repo}:*"

Make sure to reapply that permission if re-creating lambda from aws cli command

Be sure to add environment variables

aws lambda create-function \
 --function-name $LAMBDA_FUNCTION_EC2_OKOK \
 --zip-file fileb://typescript/lambda-function.zip \
 --handler "index.handler" \
 --runtime nodejs18.x \
 --role $roleArn \
 --environment "Variables={project_repo=\"$project_repo\",REGION=\"$REGION\",LAMBDA_FUNCTION_EC2_OKOK=\"$LAMBDA_FUNCTION_EC2_OKOK\", BUCKET_NAME=\"$BUCKET_NAME\",DOCKER_VOLUME_PATH=\"$DOCKER_VOLUME_PATH\"}" \
 --query "FunctionArn"