Automate EC2 Instance Shutdown with Lambdas to Minimize AWS Costs

In the cloud, cost control is a task unto itself. It takes diligent oversight over the services in your AWS cloud to ensure the monthly spend doesn't grow out of control. One method for lowering costs is to shutdown EC2 Instances when they are not in use. In this two-part series, I will show you how to reduce your monthly spend by scheduling the shut down of EC2 Instances using two different methods.

Controlling Costs

There are several factors which determine how an EC2 Instance is billed. The main factor is by usage, typically measured by the hour. As long as an EC2 Instance is in the running state, you are accruing fees, even if you are not actually using the VM. An effective method for controlling these costs is to shut down the VM when it is not needed. When the instance is in the stopped or terminated state you still pay for storage, but that is a fraction of the compute costs you pay while the instance is running.

Development, QA, or Test environments are all good use cases for automating the shutdown of an EC2 Instance. In these environments, the instances are most active during business hours. Those instances do not need to be running once everyone signs off at the end of the day.

I use two methods for shutting down EC2 instances. The first involves using CloudWatch Event Rules together with Lambdas. The second uses Launch Templates in conjunction with Auto Scaling Groups (ASG).

Each method has its own use cases. For simple startup/shutdown scheduling I like the CloudWatch/Lambda solution. This method is the easier of the two to setup. It works well when scheduling a set of EC2 instances. It is also appropriate for use cases involving EC2 instances with static public IP addresses (EIPs).

The relatively more complex Launch Template/ASG method provides additional functionality beyond scheduling like email notifications, and the ability to continually retry starting an instance if it fails to start. This is a good solution when no EIPs are involved, when all of your EC2 instances are the same instance type, or you want more robust monitoring and notifications.

I'll walk through both methods for controlling and minimizing cloud costs. This post focuses on using CloudWatch Event Rules and Lambdas to schedule instance startup and shutdown. The second post walks through using Launch Templates with ASG to achieve the same result.

CloudWatch Event Rules and Lambda

Among the many AWS services, CloudWatch Events and Lambdas can be combined to provide basic scheduling capabilities. The AWS documentation has in-depth instructions for using CloudWatch Events and Lambdas to shutdown and startup EC2 instances on a schedule.

There are three basic steps to using this technique. The steps below use the AWS Console to setup and configure the policy and role, the CloudWatch Events Rules and Lambda functions.

1. Create a custom Identity and Access Management (IAM) policy and execution role for the Lambda functions.

2. Create and test two Lambda functions; one to stop EC2 instances, and another to start them.

3. Create CloudWatch Event Rules that trigger the Lambda functions on a schedule.

Step 1 - Create IAM Policy and Role

Using the IAM service console, create an IAM policy by clicking on the "Create policy" button. This starts the policy creation wizard. On the JSON tab, enter the policy permissions from Listing 1 below. These permissions allow CloudWatch logging and EC2 startup and shutdown.

Listing 1

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "arn:aws:logs:*:*:*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ec2:Start*",
                "ec2:Stop*"
            ],
            "Resource": "*"
        }
    ]
}

Click through the wizard until you get to the "Review" page. On the Review page, enter "lambda-policy-for-scheduling-ec2-instances" as the policy name. Click the "Create policy" button to save the policy. The following screenshot shows the policy review screen. Notice that the Summary section shows permissions being applied to CloudWatch Logs and EC2 services.

To setup the role, navigate to the Roles link in the IAM Console and click the "Create role" button. This starts the role creation wizard. On the first page, select the AWS service option and choose the "Lambda" use case. Click the "Next: Permissions" button. Search for the policy created earlier named "lambda-policy-for-scheduling-ec2-instances", and select it from the list to attach it to the role. Click through the wizard until you get to the Review Role step. The following screenshot shows the role review screen.

Enter "lambda-role-for-scheduling-ec2-instances" for the role name, then click the "Create role" button to save the role.

Step 2 - Create two Lambda functions

In the AWS console, open the Lambdas service console. Click the "Create Function" button to start the wizard. Select the Author from scratch option. Name the function "start-ec2-instances". Select "python 3.9" from the Runtime dropdown. Click on the "Change default execution role" link and select the Use an existing role option. Select the role created above, "lambda-role-for-scheduling-ec2-instances" from the drop down. Leave all other settings as their defaults, as shown below.

Click the "Create Function" button to save the function. The next step in the wizard is to provide code for the lambda. The python script in Listing 2 below is used to start your EC2 instances. Copy/paste the script into the lambda wizard's code tab.

Listing 2

import boto3
region = '<your region here>'
instances = ['<your 1st instance id>', '<your 2nd instance id>']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.start_instances(InstanceIds=instances)
    print('started your instances: ' + str(instances))

The script contains two variables, region and instances. Replace <your region here> with the AWS region name where your EC2 instances are running, i.e. us-east-1. The instances variable is a comma separated list of EC2 Instance IDs. Provide one or more instances in the list. Each instance provided will be affected by the function. Navigate to the AWS EC2 console to find your Instance IDs.

After editing the script, click the "Test" button. This opens a dialog allowing the Lambda to be tested. Provide a value for the test's Event Name and click "Create". Click the "Test" button again to run the test. Review the output, looking for errors. The following screenshot shows the completed Test dialog form.

Of note; testing the Lambda function will execute the python script. Any instances listed in the startup function will be started. The same goes for the shutdown function. Testing that function will shutdown the instances.

Once any errors and warnings are resolved, repeat the process to create the "stop-ec2-instances" Lambda. The python script in Listing 3 provides the code for stopping EC2 instances. Provide the same values for region and instances here as you did for the "start-ec2-instances" script.

Listing 3

import boto3
region = '<your region here>'
instances = ['<your 1st instance id>', '<your 2nd instance id>']
ec2 = boto3.client('ec2', region_name=region)

def lambda_handler(event, context):
    ec2.stop_instances(InstanceIds=instances)
    print('stopped your instances: ' + str(instances))

Step 3 - Create CloudWatch Event Rules

The final step in this process is to setup the schedules that will stop and start the instances. Begin by navigating to the CloudWatch service in the AWS console. Click the Rules link and then the "Create rule" button. This will create a new rule which will be used to schedule instance startup.

The screenshot below shows the Event Source section. This is where the schedule is defined. Start by selecting the Schedule option and the Cron expression option. Enter the following cron expression to schedule instance startup to occur at 7 AM EST, Monday - Friday: 0 11 ? * MON-FRI *

Next, click the "Add Target" button. Select Lambda function from the drop down and select the "start-ec2-instances" function. Click the "Configure details" button and provide a name for the rule. Click the "Create rule" button to save the rule.

Repeat this process to configure a rule to shutdown the EC2 instances. The following cron expression schedules instance shutdown for 8:05 PM EST, Tuesday - Saturday: 05 00 ? * TUE-SAT *

Logging

You may have noticed earlier that the policy contained permissions for logging. Every time one of the Lambda functions executes, it logs its output to a CloudWatch Log group. To view the logs, navigate to the "Log groups" link under the CloudWatch console. In the list of log groups, you should see one named "aws/lambda/start-ec2-instances". This was created when you tested the lambda function. Click that log group and navigate to the log streams where you will find the results of the lambda's execution, including the output from any print statements in the python script.

Conclusion

Controlling costs in the cloud is a full time job. The combined power of CloudWatch Event Rules and Lambdas provides one method of controlling those costs by scheduling the startup and shutdown of EC2 instances. In the next post, we'll look at another approach to cost cutting using Launch Templates and Auto Scaling Groups. See you there.