Invoking Private API Gateway Endpoints From Step Functions

At Re:Invent 2024, AWS announced EventBridge and Step Functions integration with private APIs. Thanks to this new feature, customers can now directly invoke APIs that are inside a private VPC from EventBridge (with API destinations), or Step Functions (HTTP Tasks). Before, users had to use Lambda functions inside the VPC as a proxy to their private APIs.

In a previous post, I explained how to invoke HTTP endpoints from Step Functions with the CDK. In this issue, I will cover calling a private API Gateway endpoint using the new integration.

Overview

First, let’s examine how this new integration works and how the different components interact with each other. At the end of this post, I’ll show you how to deploy this setup with the CDK.

Here is a diagram that describes the architecture.

Source

The new private API integration is powered by VPC Lattice and AWS PrivateLink. VPC Lattice has two new features that make connecting Step Functions to a VPC possible: Resource Gateways and Resource Configurations

Resource Gateway

A resource gateway is a point of entry into the VPC where your resources reside. It can span one or more availability zones through the VPC subnets.

To access a private API Gateway from Step Functions, we need a Resource Gateway that lives in the same VPC and subnets as the VPC endpoint that is attached to the API.

Resource Configuration

Once we have a Resource Gateway for our VPC, we can create and attach Resource Configurations to it. Resource Configurations represent resources that are accessible through the gateway, and how they are accessed.

In the case of API Gateway, a resource configuration consists of the VPC endpoint’s regional DNS name. We can also specify a port or range of ports that are accessible, which in our case is just 443 (for HTTPS).

EventBridge Connection

To call HTTP endpoints, Step Functions uses EventBridge Connections. The connection defines the authorization method, and the credentials to access the endpoint. Connections now have a new capability that allows integration with private APIs through a VPC Lattice Resource Configuration.

For API Gateway, the Resource Configuration is the one that defines the API Gateway.

And that’s all we need. Everything else works the same as calling a public HTTP endpoint.

Definition With the CDK

As explained earlier, we need a Resource Gateway that serves as the point of ingress into our VPC. We also create a security group that only allows egress to port 443, which is all we need for this use case:

const rgSecurityGroup = new SecurityGroup(this, 'ResourceGatewaySG', {
  vpc: vpc,
  allowAllOutbound: false,
});

rgSecurityGroup.addEgressRule(
  Peer.ipv4(vpc.vpcCidrBlock),
  Port.tcp(443),
  'Allow HTTPS traffic from Resource Gateway',
);

// Resource Gateway
const resourceGateway = new CfnResourceGateway(this, 'ResourceGateway', {
  name: 'private-api-access',
  ipAddressType: 'IPV4',
  vpcIdentifier: vpc.vpcId, 
  subnetIds: vpc.isolatedSubnets.map((subnet) => subnet.subnetId), // all isolated subnets
  securityGroupIds: [rgSecurityGroup.securityGroupId],
});

We also need a Resource Config that describes the API Gateway’s VPC endpoint.

// Resource Configuration
const resourceConfig = new CfnResourceConfiguration(
  this,
  'ResourceConfig',
  {
    name: 'sf-private-api',
    portRanges: ['443'],
    resourceGatewayId: resourceGateway.ref,
    resourceConfigurationType: 'SINGLE',
  },
);

resourceConfig.addPropertyOverride(
  'ResourceConfigurationDefinition.DnsResource',
  {
    DomainName: Fn.select(
      1,
      Fn.split(':', Fn.select(0, api.vpcEndpoint.vpcEndpointDnsEntries)),
    ),
    IpAddressType: 'IPV4',
  },
);

At the time of writing, the CfnResourceConfiguration L1 construct does not support DnsResource for ResourceConfigurationDefinition, so I’m using an override. For DomainName, we need the regional public DNS name of the VPC endpoint of the API Gateway, which is the first item of the DnsEntries CloudFormation returned value. It’s prefixed with the hosted zone id, so I’m using intrinsic functions to extract the value.

We can now use the configuration in our Step Functions definition:

const connection = new Connection(this, 'ApiConnection', {
  authorization: Authorization.apiKey(
    'x-api-key',
    SecretValue.unsafePlainText('demo'),
  ),
});

(connection.node.children[0] as CfnConnection).addPropertyOverride(
  'InvocationConnectivityParameters',
  {
    ResourceParameters: {
      ResourceConfigurationArn: resourceConfig.attrArn,
    },
  },
);

const http = new HttpInvoke(this, 'Http', {
  apiRoot: api.url, // url of the API Gateway
  apiEndpoint: TaskInput.fromText(`hello`),
  method: TaskInput.fromText('GET'),
  connection: connection,
});

The Connection construct does not support InvocationConnectivityParameters yet, so I’m also using an override here as well.

And here you have it! You can find this code in full on GitHub.

Conclusion

With this new integration with Amazon VPC Lattice and AWS PrivateLink, teams now have the ability to invoke private API Gateway endpoints directly from AWS Step Functions or Amazon EventBridge. This eliminates the need for a Lambda Function, which in turn reduces the amount of code required and decreases overhead. It's a significant step forward for teams looking to optimize and simplify their cloud infrastructure.