During AWS re:Invent 2021, David Brown, Amazon EC2 VP, announced Amazon VPC IP Address Manager (IPAM).
Why VPC IPAM?
VPCs with overlapping CIDRs cannot be peered, and is usually resolved only by destroying and recreating the VPC with the correct CIDR. If there are many existing workloads in the VPC, this process will incur painful migration effort and risks to live production systems.
IP address allocation to Amazon VPCs is something that organizations want to get correct at the beginning and never have to think about in the future. Amazon VPC IP Address Manager (VPC IPAM) helps by handling the IP address allocation, ensuring an organization will never issue conflicting CIDRs to its VPCs.
VPC IPAM can also save organizations from future embarrassments when new AWS resources are blocked from scaling due to a lack of free IP addresses. VPC IPAM has the ability to monitor IP address utilization, generating alerts to administrators when certain thresholds are crossed.
AWS VPC IPAM basics
[1] Service-Linked Role and IPAM Administration
The service-linked role AWSServiceRoleForIPAM is required for IPAM to discover, monitor and manage IPAM resources (resources are covered at the end of this list).
For IPAM administration, it slightly differs depending on situation:
- Using AWS Organizations (documentation)
- Delegate an Organizations member account to be the IPAM account.
- The IPAM account will then be responsible for making IPAM-level configurations.
- The IPAM service-linked role will be automatically created in each member account in your organization.
For single AWS account (no AWS Organizations), the IPAM service-linked role needs to be created manually.
[2] IPAM
Before everything else, you'd need to create an IPAM first (documentation). At this point of time, you have to decide IPAM's operating regions. This ringfences IPAM to the regions where you would have resources for IPAM to manage. This can be modified later on.
[3] Scope (Private & Public)
After creating an IPAM, 2 scopes are automatically created, 1 public and 1 private scope. To explain what scopes are, I've compiled some findings from the documentation, re:Invent and my own testing:
- You can only create private scopes.
- Public scopes seem to be for your BYOIP (bring your own IP) resources. These IP addresses don't belong to AWS and are not managed by IPAM.
- All IPAM resources (non-BYOIP) will be in the automatically-created private scope by default. You can shift them to a different scope subsequently.
- Each private scope contains it's own set of managed IP CIDRs, IPAM Pools and resources. Inheriting overlapping AWS resources with overlapping IP CIDRS is a consideration for having additional private scopes.
[4] Pool (Top-Level)
From the documentation, a pool is a collection of contiguous IP address ranges (or CIDRs). When creating a top-level pool, it should contain all the specific CIDRs you'd like to allocate to resources within a scope.
AWS doesn't let you define regions here. So the CIDRs here can be consumed in any region.
Note: For each top-level pool, you can set allocation rules to enforce IPAM resource tagging and network mask allocation.
[5] Pool (Sub-Level)
A sub-level pool can be restricted to a specific region and is allocated a CIDR from a higher level pool. A recommended pool structure from AWS looks something like this:
You can have many levels of pools. Design the pool structure according to your needs. Sub-level pools do not require specific CIDRs to be defined; you could define a network mask and let IPAM handle the CIDR allocation automatically.
Note: For each sub-level pool, you can set allocation rules to enforce IPAM resource tagging, network mask allocation and region.
[6] Pool sharing and IP allocation
If you are using AWS Organizations, IPAM pools can be shared to member accounts via AWS Resource Access Manager (RAM).
Once shared, IPAM resources being created can utilize the pool to receive allocation of CIDRs from the pool. Additionally, a user from the member account can also be allocated an IP address through an API call to the shared pool.
[7] IPAM managed and monitored Resources
From the documentation:
In IPAM, a resource is an AWS service entity that is assigned an IP address or CIDR block. IPAM manages some resources, but only monitors other resources.
A managed resource has CIDR allocated from an IPAM pool, and is typically a VPC. A monitored resource extends beyond that, covering:
- VPCs
- Subnets
- Elastic IPs
- Subnet reserves
- Public IPv4 pools
Note: The number of "active" IP addresses determines IPAM's costs. See the next section for more information on this.
That wraps up the Amazon VPC IP Address Manager basics and its various components.
You should be careful of...
In this section, I'll share some of my findings based on my experience testing the service out without AWS Organizations.
- It is possible to create your IPAM, scopes, pools and allocate CIDRs to VPCs before creating your service-linked role. You wouldn't be able to monitor what you have allocated.
- Once a pool locale/region is configured, it cannot be changed. The only way is to recreate it with the correct locale/region.
- You cannot delete a higher level pool without deleting it's sub-levels first.
- You must deprovision all CIDRs in a pool before you can delete your pool.
Costs
From the VPC pricing page:
You pay an hourly rate for each active IP address that you manage using IP Address Manager (IPAM). An active IP address is defined as an IP address assigned to a resource such as an EC2 instance or an Elastic Network Interface (ENI).
It is important to note that not only IPAM-managed resources with active IP addresses are billed, IPAM-monitored resources, with IP addresses not allocated by IPAM are billed as well. In Singapore (ap-southeast-1), each active IP address costs $0.00027 USD per hour.
This is a problem for users who run a self-managed Kubernetes cluster in AWS or use Amazon Elastic Kubernetes Service (EKS) with self-managed nodes. These Kubernetes nodes have many IP addresses per Elastic Network Interface (ENI) and with each IP address being classified as an "active IP address" to IPAM, it's easy to see how your bill could skyrocket.
Fortunately, I received a great tip to mitigate this from the AWS Community Builders group. By using prefix delegation, IPAM counts the whole delegation (16 IP addresses) as a single IP address. You can find additional considerations and limitations for using prefixes in the documentation.
Alternatively, you can consider using a separate CIDR as an overlay and potentially reserving those CIDRs manually on VPC IPAM. Since these CIDRs are not assigned on an ENI, it does not count as an active IP address.
Closing Thoughts & Whether You Should use VPC IPAM
Amazon VPC IPAM is a compelling and useful service for a moderately-sized AWS Organizations greenfield or brownfield set up. If you are running a Kubernetes cluster with self-managed nodes, VPC IPAM's costs could be exponentially higher. Consider some of the above suggestions to reduce costs before using the service.
Stay tuned for a future blogpost for some sample Terraform code to get you started on VPC IPAM.
Top comments (2)
Excellent blog post! Love the way you summarized all the terms. One small thing I noticed:
You can create pools in either scope type (public or private). Here is an example of a pool in a public scope in the terraform provider acceptance tests. However, you can only make scopes that are type private
Thanks for pointing that out! Will make some amendments.