Release Dangling Elastic IPs using Lambda and Go SDK

Introduction

Today we are going to do a hands-on on how to kill a dangling EIP or in other words, deallocate an unassigned Elastic IP. But before that, let us know...

What is an Elastic IP and why do we need it?

Elastic IPs are simply IP addresses provided by Amazon AWS. These are usually attached to internet gateways such as load balancers, or even EC2 instances. I have been using it with EC2 instances directly for my testing use case.

Normally when you configure your EC2 instance to have a public IP but at the same time, you don't use EIP. The problem is that when you stop and start your instance, your IP address is changed.

This is where Elastic IP shines. You attach an EIP to the EC2 instance and when you stop and start your instance, your IP won't be changed.

This is extremely important if the EC2 you are running is hosting an internet-facing web server. Web servers shouldn't generally frequently change their IPs.

But why do we need an EIP killer?

There is 1 good thing about EIP and 1 bad thing. The good thing is EIP is free to use. You don't need to pay anything for the static IP you are using. How cool!

The bad thing is, that when EIP is not attached to an instance, it cost you money. How warm!

It won't cost you much if you leave it detached for a few hours. But as a human, we tend to forget if EIP is in use and only get to know about it when we get our monthly bill.

In the Asia Pacific (Mumbai) region, if I forget to kill an EIP for a month, I get around a $4 bill. This was my main motivation to write this article.

With an EIP killer, I will be able to use Elastic IP with confidence by releasing it when not in use.

Get Started

What we'll work with:

• AWS Lambda
• Go
• AWS Go SDK
• AWS CLI

What we're going to do:

• Run a Lambda on a predefined interval.
• This Lambda will use AWS SDK to check if any dangling EIP is lying around.
• If dangling EIP is found, deallocate, or in our language, kill it.

Initially, I was going to use Python for the Lambda runtime, but Go is faster than Python. We are going to benefit from this decision as we can run our function for a smaller time span. This is important because we plan to run our Lambda at frequent intervals.

With this post, I mostly plan to expand my knowledge in the field of serverless computing. I intend to touch Lambda, EventBridge, and related constructs.

Phase 1: Provision a Lambda function

The main task in this phase is to manually create a function. In the second phase, we'll see what is the logic for killing the EIPs. Let's get started.

1. Go through the web UI and choose to create a function.
2. Keep "Author from scratch" selected.
3. Fill in the name. I liked to call it "eip-killer".
4. Select the runtime to be Go 1.x.
5. Keep everything else on default. Hit Create function.

Note: Creating Lambda with default settings also creates an IAM role. This IAM role dictates how this Lambda interacts with other AWS services. By default, this role allows Lambda to talk with CloudWatch to put logs. You can add more policies to this role when working with other AWS services.

Let's go ahead and do a quick invoke to test if the function is working.

Our EIP-killer is alive. But doesn't know how to kill. Let's teach it in the next phase.

Phase 2: Write the EIP killing logic

While testing, you might have got a view of the Code tab of the function which says, "The code editor does not support the Go 1.x runtime". What this means for us developers is that we have to use our text editor to write the logic and upload the pre-built binary to Lambda.

The prime documentation for Go AWS SDK is https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html, but don't worry. I'll filter out the required material required for this post.

Phase 2.1: The initial deployment process

Before we write the actual logic, let's write a simple "Hello EIP Killer" code to get the awareness of the process required for uploading the code.

1. Create a workspace for our lambda code.

mkdir eip-killer
cd eip-killer

1. Initialize a go module in the folder.

go mod init EIP-killer

You could also put the GitHub/GitLab URL of the repo where you intend to put your lambda code at.

1. Create a main.go and put this code into it:

package main

import (
    "github.com/aws/aws-lambda-go/lambda"
)

func hello() (string, error) {
    return "Hello EIP Killer!", nil
}

func main() {
    // Make the handler available for Remote Procedure Call by AWS Lambda
    lambda.Start(hello)
}

As we are using external dependency, get it with this command:

go get github.com/aws/aws-lambda-go/lambda

1. Build a binary out of main.go:

GOOS=linux GOARCH=amd64 go build -o main main.go

Remember while creating the function we had selected architecture to be x86_64. So we need to reflect that with GOARCH=amd64.

1. Zip the binary. Upload the zip to function.

zip main.zip main

Upload using this command:

aws lambda update-function-code --function-name eip-killer --zip-file fileb://main.zip

While this could have been enough, the default handler (where lambda begins executing) in function with factory settings is hello. But our code uses main. So we need to update the handler config to match that.

We need this update only once.

aws lambda update-function-configuration --function-name eip-killer --handler main

After this, when you run the func:

You can roam around this path: https://github.com/aws/aws-lambda-go if you want to dig deeper into Go+Lambda. But for now, I'm going to use AWS SDK to do our magic.

Phase 2.2: Logic to deallocate unused EIPs

Open up the main.go again and write this:

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/aws/aws-lambda-go/lambda"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/ec2"
    "github.com/aws/aws-sdk-go-v2/service/ec2/types"
)

type MyEvent struct {
    Name string `json:"name"`
}

func Handler(ctx context.Context, name MyEvent) (string, error) {
    // Load the Shared AWS Configuration (~/.aws/config)
    cfg, err := config.LoadDefaultConfig(ctx)
    if err != nil {
        log.Fatal(err)
    }

    // Create an Amazon EC2 service client
    ec2ServiceClient := ec2.NewFromConfig(cfg)

    // DescribeAddressesInput with no filter means list all addresses
    IpListFilter := &ec2.DescribeAddressesInput{}

    // Ask the EC2 client to list all Elastic IPs.
    result, err := ec2ServiceClient.DescribeAddresses(context.TODO(), IpListFilter)
    if err != nil {
        fmt.Println("Got an error retrieving information about your Amazon Elastic IPs:")
        return "", err
    }

    // Out of all IPs, check if they have an allocation ID. If not, they are a candidate for deletion
    for _, a := range result.Addresses {
        fmt.Println("Allocation ID: " + *a.AllocationId)
        fmt.Println("Public IP: " + *a.PublicIp)

        // if this EIP is not associated with any AWS resources, proceed with killing.
        if a.AssociationId == nil {
            // extra step for flexibility
            if isThisEIPKillable(a.Tags) {
                fmt.Println("About to be killed")
                // Release/kill the Elastic IP
                ReleaseAddressFilter := &ec2.ReleaseAddressInput{AllocationId: a.AllocationId}
                ec2ServiceClient.ReleaseAddress(context.TODO(), ReleaseAddressFilter)
                fmt.Println(*a.PublicIp + " is not more")
            } else {
                fmt.Println(*a.PublicIp + " has KILLERHOLD enabled and is going to stay")
            }

        } else {
            fmt.Println("Association ID: " + *a.AssociationId)
            fmt.Println("Instance ID: " + *a.InstanceId)
            fmt.Println("PrivateIpAddress: " + *a.PrivateIpAddress)
        }

        fmt.Println("")
    }
    return "Done execution", nil
}

// isThisEIPKillable scans to tags of Elastic IP and only returns false when
// EIPKILLER is tag is set to true.
func isThisEIPKillable(tags []types.Tag) bool {
    for _, tag := range tags {
        if *tag.Key == "KILLERHOLD" && *tag.Value == "true" {
            return false
        }
    }
    return true
}

func main() {
    lambda.Start(Handler)
}

Quite a large source code. Let us understand it line by line.

Line 14-16 declares an event struct. This struct is going to be a parameter to the Handler function.

On line 18, we see function with signature Handler(ctx context.Context, name MyEvent) (string, error). Handler here is the name of the function.

The first parameter to the handler is the context object. If you are working with web services in golang, you must be aware of that.

Next is our MyEvent which is defined above. This is as per Lambda's spec.

Our handler function also returns a string and an error. Which is then bubbled to Lambda logs.

Line 20 deals with configuration. Please note that we are using AWS SDK. And the SDK can run on platforms other than Lambda. On the local system, the default config is located at ~/.aws/config. But on Lambda, it automatically loads the region it's running in.

On line 26 we create a new EC2 client to interact with our AWS account. We initialize this client with the config instance we created on line 15.

On line 29 we have initialized a filter for IPs. This seems to be an added step. But this variable is going to be a mandatory parameter for our next command.

On line 32 we get a list of all Elastic IPs associated with our account. There are no filters to the result as we have not specified any filter on line 29.

It's time to look at the isThisEIPKillable function which is from 69 to 76. This is an extra feature for flexibility. This will forgive any EIP which has the tag KILLERHOLD set to true. So when you have an EIP you don't want to kill, you set this tag to it. This is good for the experiment as our Lambda will be running on a 1 hours (configurable by user) interval and it might not be desired outcome to delete them every 1 hour.

Let us get back to our main flow. We have a result variable with all the addresses in it. On line 40 we check if the IP is associated with any EC2 or not. Then we run the helper function we just discussed. If all is okay, meaning that KIP can be released, we go ahead and release the address back to the AWS pool.

On lines 57-59, if EIP is associated with any EC2, we print some metadata about the EIP association.

That's it. It was not that hard to understand, isn't it? Before we deploy it, let's create some EIP and not allocate it to any EC2. We'll also have some EIP having KILLERHOLD set to true.

Phase 2.3: Test our EIP killer Give Lambda permission to release and read EIPs

I hope you have updated your lambda function and pushed the zip file to the function. Review Phase 2.1 if you have not. Continue reading if you did.

Before I test it with actual EIPs, I wanted to show something else. Right now if you try to test the function, you'd see something like this:

If you clearly read the error message, it says that API call returned 403. Meaning that Lambda was not authorized to make the call to EIP APIs.

To overcome this problem, we need to give access to Lambda to read some metadata from EC2.

When no Lambda Web UI for the function. If you go to Configuration > Permission > Execution role as described in the image below. You'll see a role that was created while we created the function.

Click on that and you'll be taken to the IAM page for that role.

When on that page, look for a dropdown named Add permissions and click on Create inline policy.

Switch from Visual editor to JSON. And paste this policy.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "ec2:ReleaseAddress",
                "ec2:DescribeAddresses"
            ],
            "Resource": "*"
        }
    ]
}

As you can see, in the Action section, we have added permission for ec2:ReleaseAddress, and ec2:DescribeAddresses.

Review policy and give it a name, and then Create policy.

When done, you'd see this policy in the list of policies attached to the role.

Phase 2.3: Test our EIP killer

We are not set for the actual testing.

We are going to allocate 3 Elastic IPs for our testing purpose.

One we'll attach to the EC2 instance.

The second will be dangling.

And the third one will be also dangling but will have our KILLERHOLD set to true.

To actually invoke the function (apart from testing it), one way is to create a public URL. We can also invoke through the AWS CLI, but I'm not choosing that for now. You can do it.

Then I went ahead and opened that URL to execute the function.

You can configure AWS API Gateway in front of the function. But that's out of the scope of this post.

Next, we see the result of the invocation.

That's the end of Phase 2. We can kill EIPs now. How wonderful.

Phase 3: Trigger Lambda at a 1-hour interval

The main part of the EIP killer is already developed. Let's now execute it at a 1-hour interval as planned before.

Phase 3.1: Configure EventBridge aka CloudWatch Events

We are going to emit an event at our 1-hour interval and we are going to use EventBridge for this.

I have recorded a screencast gif to preserve some space on this post.

Once done, this event should appear in the Trigger section of the Lambda function; as seen in the below image

Phase 3.2: Create an EIP and go to sleep

As I'm writing this post at 4 am. I'm going to create an EIP and go to sleep. Will check in the morning if our setup is working.

A few hours later...

I woke up this morning with only 1 EIP.

And this is how we gonna achieve this killing at 1 hour. You can configure the interval, but I think this is optimal. You also don't need to have a public IP for this. So if you have, you can delete it now from the web interface.