User-Centric CloudFormation Templates

This topic is the subject of my presentation at the AWS User Group Mega Manila Meetup last August 20, 2020. The customizability part of my presentation is covered in this post

CloudFormation templates are the least likely of all your code to change once it has been deployed to production. For most small to medium-sized teams, one AWS-savvy person in the group will create the CF template, and it will barely change throughout its lifetime. Since just one person is working on the CF template, there's little incentive to make the template user-friendly. This goes to bite the team when the AWS engineer leaves the company, and those left behind have to update the template in an emergency.

Design CF templates for technical people

The key to addressing this issue is taking the time to improve the user experience. It's probably true that it's a fellow developer who will make changes to the template. We don't have to simplify the template to be appreciated by the common man. We just have to make sure that someone technical with a slight idea of how AWS works can understand the template enough to make changes to it.

From my experience of making CF templates for clients, they usually are somewhat versed in CloudFormation. But of course, they'd want the CF template to be easy to use and maintain. Based on their change requests, I identified three things they usually look for in a template:

• 📚 Readable - can the user know what the CF template is doing just by reading it?
• 🎛 Customizable - how much can the user change with the stack without changing the template itself?
• 🕹 Ease of use - how easy is your template to use?

In this post, I'll present six tricks I do to make the CF template more user-centric and make my clients happy.

(1) Divide your parameters into sections

When you want your templates to be customizable, you'll inevitably end up with way too many parameters. To give you an idea, for the ECS service CF template that I created, I had 29 parameters. Unfortunately for CloudFormation, we cannot add HTML/CSS stylings to divide the 29 parameters into their respective sections: It's just presented as one alphabetically-ordered long list of parameters. This can be overwhelming for most users trying to make use of your template for the first time.

SOLUTION: I prepend the parameter's name based on their section. For the ECS Service template, I have three features (i.e AutoScaling, ServiceDiscovery, ALBConnection), each requiring their own parameters. So I grouped the parameters by prepending the section to the name of each parameter so that when it shows up on CloudFormation, parameters within the same section will be lumped together.

(2) Give your users a choice whether to enable a feature or not

Often, the user will want a choice whether to use a feature in your template or not. A way to do this is by creating multiple templates: "ecs-service-with-auto-scaling", "ecs-service-no-auto-scaling". But this means your code is repeated across templates. This arrangement would be tough to maintain as the number of feature combinations increases.

SOLUTION

• You can have "switch" parameters where you give the user an option to use the feature or not. I usually add "AA" after the section's prefix to make sure the "switch" parameters always end up on top of the section. For the ECS example above, this could mean the switch parameter's name would be: "AutoScalingAASwitch"
• If the feature is not going to be used, the rest of the settings should not be filled up anymore. To add a signifier for this, I prefix the parameter's description with the label "REQUIRED FOR THE SPECIFIC FEATURE TO WORK."
• In the template itself, I add conditions on whether to create a specific service or not. In the template snippet below, the ECSInstanceProfile resource is only created if Ec2InstanceProfileAASwitch = "Yes, add an instance profile".
  • The IamInstanceProfile property of the Ec2Instance still references the ECSInstanceProfile even if it is not created. To remedy this, we added an if statement !If [AddInstanceProfile, !Ref ECSInstanceProfile, !Ref "AWS::NoValue"] to point the property IamInstanceProfile to NoValue if the ECSInstanceProfile was not created.

AWSTemplateFormatVersion: '2010-09-09'

Parameters: 
  Ec2InstanceProfileAASwitch:
    Description: UNIVERSALLY REQUIRED - Will your Ec2 have an instance profile?
    Default: No, it will not have an instance profile
    Type: String
    AllowedValues:
      - Yes, add an instance profile
      - No, it will not have an instance profile

Conditions:
  AddInstanceProfile: !Equals [!Ref Ec2InstanceProfileAASwitch, "Yes, add an instance profile"]
  NoInstanceProfile: !Equals [!Ref Ec2InstanceProfileAASwitch, "No, it will not have an instance profile"]

Resources:
  ECSInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Condition: AddInstanceProfile
    DependsOn:
      - ECSRole
    Properties:
      Path: /
      Roles:
        - !Ref ECSRole

  Ec2Instance:
    Type: AWS::EC2::Instance
    Properties:
      ImageId: ami-1234567891011
      KeyName:
        Ref: BastionHostKeyName
      InstanceType:
        Ref: BastionHostInstanceType
      IamInstanceProfile: !If [AddInstanceProfile, !Ref ECSInstanceProfile, !Ref "AWS::NoValue"]
  ECSRole:
    ...

(3) Use CF Macros to reduce the number of parameters

As you build more customizable templates, you will stumble upon CloudFormation's limitations. One of them is that you have to create dozens of parameters when you want your templates to be customizable. But what if there's a way to group similar parameters?

Parameters:
  SecurityGroupIngressRuleString:
    Description: REQUIRED - Required for the Security Group of the ECS Service. Make sure to allow access to the assigned SSH Port.
    Type: CommaDelimitedList
    MinLength: '9'
    Default: 80:80:0.0.0.0/0,443:443:sg-1234567890

Resources:
  SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      VpcId:
        Ref: VpcId
      GroupDescription: Security Group
      SecurityGroupIngress: |
        #!PyPlate
        output = []
        for ip_combo in params['SecurityGroupIngressRuleString']:
          from_port, to_port, cidr_ip = ip_combo.split(':')
          if 'sg' in cidr_ip:
            output.append({'IpProtocol': 'tcp', 'FromPort': from_port, 'ToPort': to_port, 'SourceSecurityGroupId': cidr_ip})
          else:
            output.append({'IpProtocol': 'tcp', 'FromPort': from_port, 'ToPort': to_port, 'CidrIp': cidr_ip})

In the code snippet above, we have the SecurityGroupIngressRuleString parameter. It is a comma-separated list that specifies the ingress rules for the security group we are creating. Each element of the string contain one ingress rule with the format fromRange:toRange:Destination:

• The "from range" and "to range" are the range of ports we want to open to the destination.
• The "destination" can either be an IP CIDR (12.33.44.55/32, for example) or a security group ID.

Hence, the statement1000:2000:sg-1234567890 means that we are allowing all resources associated with the sg-1234567890 security group to communicate with the resources associated with this security group via any ports between 1000 and 2000. The user can add as many ingress rules as he likes inside that one parameter.

How does it all work?

We first register the Pyplate Macro by uploading this CF template. This template creates a Lambda function.

When we upload a CF template, CloudFormation looks at the transforms section. Since we added PyPlate in our template snippet above, it finds the PyPlate CloudFormation Macro that we just registered. It triggers the Lambda function and passes the template and our runtime parameters to that function. PyPlate looks at our template line by line and when it sees the "#!PyPlate" string, it runs the Python code. Whatever the variable "output" contains becomes the output for that block of the CF template.

Taking a look at the Python section of the code snippet, we loop through each ingress rule in the params['SecurityGroupIngressRuleString'] list, create a dictionary, and append that to the output variable.

Take it further

You can also create a key-pair style parameter (i.e. "MinCount=1,MaxCount=5"). You can then create the necessary Python code so that MinCount and MaxCount property of a resource will use the respective value in your parameter. With this, you needed to have one parameter instead of 2. (Look at the image at #4 for an illustration).

(4) Create sensible default parameters

Even with all your best effort to reduce the number of parameters the user has to input, sometimes it will not be enough. An alternative is to provide standard defaults to these parameters. The user doesn't have to change these defaults if they aren't particularly sensitive about how the feature works. Maybe they just want it to work out of the box?

An example would be the Auto Scaling feature of the ECS Service template I'm building. Imagine if I left this part blank. The user will have to decide each parameter by himself. By having a default, the user can make the Auto Scaling feature work out-of-the-box. He can return to this later if he wants to finetune the Auto Scaling behavior based on his needs.

(5) Use CF Cross-Stack References to reference values from another stack

Creating sensible defaults still leaves some parameters to be filled up. Parameters like ALB_Listener_ARN, ECS Cluster Name, VPC ID, Subnet IDs need to be filled up with the correct value.

One approach is to use Cross-Stack references. For example, we have the template of the "Networking Stack" below. In its output section, we see that we exported some values. These values are going to be available for other templates to use once this stack has been deployed. Notice how we prepended the AWS Stack Name of this template to make sure the variables are unique should we decide to deploy multiple stacks from this template.

AWSTemplateFormatVersion: '2010-09-09'
Description: 'AWS CloudFormation Sample Template from AWS'
Resources:
  VPC:
    Type: AWS::EC2::VPC
    Properties:
      EnableDnsSupport: 'true'
      EnableDnsHostnames: 'true'
      CidrBlock: 10.0.0.0/16

  PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
      VpcId:
        Ref: VPC
      CidrBlock: 10.0.0.0/24

  InternetGateway:
    Type: AWS::EC2::InternetGateway

Outputs:
  VPCId:
    Description: VPC ID
    Value:
      Ref: VPC
    Export:
      Name:
        Fn::Sub: "${AWS::StackName}-VPCID"

  PublicSubnet:
    Description: The subnet ID to use for public web servers
    Value:
      Ref: PublicSubnet
    Export:
      Name:
        Fn::Sub: "${AWS::StackName}-SubnetID"

In a second CF template, we are deploying an EC2 instance. To deploy the instance, we need some values from the Network Stack (i.e the VPC and the subnet). Instead of asking for these parameters as VPC-id and Subnet ID, we ask the name of the CF stack that created the VPC and subnet resources. Then, in the EC2 instance section, we just reference the name of the variable we exported in the "Network" Stack. We append the name of the stack to form the variable: "${NetworkStackName}-SubnetID".

AWSTemplateFormatVersion: 2010-09-09
Description: AWS CF Cross-Stack Reference Sample Template
Parameters:
  NetworkStackName:
    Description: Stack Name of the Network Stack
      stack.
    Type: String
    Default: SampleNetworkCrossStack
Resources:
  WebServerInstance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-05a66c68
      NetworkInterfaces:
      - GroupSet:
        - Fn::ImportValue:
            Fn::Sub: "${NetworkStackName}-SecurityGroupID"
        AssociatePublicIpAddress: 'true'
        DeviceIndex: '0'
        DeleteOnTermination: 'true'
        SubnetId:
          Fn::ImportValue:
            Fn::Sub: "${NetworkStackName}-SubnetID"

(6) Use a local preprocessor to create profiles

An alternative to cross-stack references is using a local preprocessor to prefill default values. I did this by creating a simple Python script in my local and running that script before uploading the template.

import cfn_flip.yaml_dumper
import yaml

from cfn_tools import load_yaml

# (1) read the template
with open("raw_template.yml") as f:
    raw = f.read()
    cf_template = load_yaml(raw)

# (2) read the profile definition
with open("blogsite-profile.yml") as f:
    raw = f.read()
    profile_definition = load_yaml(raw)

# (3) merge the values of the profile definition as the default value for its respective parameter in the template
for parameter_key in profile_definition:
    parameter_value = profile_definition[parameter_key]

    cf_template['Parameters'][parameter_key]['Default'] = parameter_value

# (4) write the new template to a new YAML file
with open("output_template.yml", 'w') as f:
    dumper = cfn_flip.yaml_dumper.get_dumper(clean_up=False, long_form=False)
    raw = yaml.dump(
        cf_template,
        Dumper=dumper,
        default_flow_style=False,
        allow_unicode=True
    )
    f.write(raw)

First, we have the CF template we use to create the resources. We also have another YAML file that contains the values we want to insert in our template's parameters (let's call this profile_definition). In the 4th code block in our Python script, we go through each parameter in the profile_definition, find the equivalent parameter for it in the cloudformation template, and set the parameter's default value equal to the value of the parameter in the profile_definition.

The process looks like this:

• You can create profile_definitions for different environments / applications you might have
• We fill in the default value of the parameter to provide a standard for the application. But the user can still change these values should he need to.

That's all!

How about you? What are the tricks you can share to make your AWS CloudFormation templates more user-centric?

I'm happy to take your comments/feedback on this post. Just comment below, or message me!

Special thanks to Daniel McCullough for the background image