One best practice with CloudWatch agent on local instances is to utilize SSM Parameter Store to organize CloudWatch agent configurations. This allows configurations to be more reusable across multiple instances. One consideration is the instance will need access to the SSM service through either an internet gateway, a NAT gateway, or a VPC Endpoint. This guide works through the process on Linux systems but should be easily modified to work on Windows as well.
IAM Profile Setup
Before CloudWatch agent can work with logs some setup is required. First an EC2 Instance Profile will need to be setup which allows for posting logs to CloudWatch. It will also need to allow reading from the appropriate parameter store location. The IAM managed policy CloudWatchAgentServerPolicy can handle this or can be used as a reference to create a more strict policy. As of writing the policy looks like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"ec2:DescribeVolumes",
"ec2:DescribeTags",
"logs:PutLogEvents",
"logs:DescribeLogStreams",
"logs:DescribeLogGroups",
"logs:CreateLogStream",
"logs:CreateLogGroup"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssm:GetParameter"
],
"Resource": "arn:aws:ssm:*:*:parameter/AmazonCloudWatch-*"
}
]
}
As the last statement indicates, only SSM parameters that start with "AmazonCloudWatch-" will be readable by the role. Other naming conventions will require a custom policy.If configs are encrypted via SecureString allowing KMS decrypt will also be required:
{
"Version": "2012-10-17",
"Statement": {
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:*:111122223333:key/MyKeyHere"
}
}
For CloudWatch agents to be managed via SSM, the AWS managed policy AmazonSSMManagedInstanceCore will provide the necessary permissions.
Cloud Watch Agent Installation
Some instances will also require CloudWatch agent installed. The AWS documentation has several methods for this, including using Systems Manager. Command line installations will also need to have the appropriate user data defined or utilize a custom image. If this seems like too much work Amazon 2 AMIs come with the agent installed by default.
Cloud Watch Agent Basic Configuration
Depending on installation methods some CloudWatch agents may not have a config file setup already. Assuming a default installation of /opt/aws/ the CloudWatch agent configuration script can generate one by default:
# sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard
Not that as is the system does not have permissions to push the config to parameter store. Permissions can either be added or simply copy the contents of the output file to parameter store manually (making sure that it follows the AmazonCloudWatch- prefix convention to work with the managed policy).
Working With Multiple Configurations
Normal Linux installations will have a directory /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d that can hold multiple configuration files. The agent level should generally be part of the main config. Then app specific settings can be declared by declaring the appropriate top level key down to the specific setting. As an example:
{
"logs": {
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log",
"log_group_name": "LogsTest",
"log_stream_name": "amazon-cloudwatch-agent.log",
"timezone": "UTC"
}
]
}
}
}
}
This will combine the core configuration with a new substructure of logs which defines an application specific logging format. After a uniquely named file is placed in the amazon-cloudwatch-agent.d directory the CloudWatch agent will need to be restarted.
Working With Parameter Store
Technically you can simply use the AWS CLI to get the parameter store contents and then put it in the file. However, AWS provides a tool to simplify this process: amazon-cloudwatch-agent-ctl. To begin, take the JSON above and put it in parameter store. You will need to either create the "LogsTest" log group or update it to another log group of choosing. Be sure the parameter store entry name starts with a AmazonCloudWatch- prefix. If you're worried about security use SecureString to encrypt (again, you will need to add the appropriate KMS decrypt permissions if you do). Once this is done, simply run:
$ /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a append-config -s -c ssm:AmazonCloudWatch-CWAgent
Be sure to replace AmazonCloudWatch-CWAgent with the name of your SSM parameter. Note that despite being in the same product, secrets manager will not work here, only parameter store entries. If the run is successful an entry prefixed with ssm_ should be in the /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d. In the event something goes wrong removal of the file along with another append-config should work until the issue is addressed.
Alternatively if you're using SSM to manage instances the AmazonCloudWatch-ManageAgent run command allows you to add configurations as well. This will once again require the appropriate network access to reach the SSM service and AmazonSSMManagedInstanceCore or similar policy to work with the document.
Conclusion
Assuming everything was setup properly the amazon-cloudwatch-agent.lg stream should appear under the appropriate log group and log data should be pushed. Be sure to look over the metrics and logs sections to see how things are laid out when customizing your configurations. I hope this article has helped those looking to centralize their configurations in parameter store to make them more reusable.
Top comments (0)