When you need websockets in a project on AWS most likely API Gateway Websockets (I will refer to it as API Gateway from now on) is the first service coming to mind. At some point when looking into options, I ran into IoT Core instead. I thought this was meant only for very specific scenarios involving hardware; however it also supports MQTT over websockets which makes it an amazing choice for web and app. I think this is a hidden gem in the AWS ecosystem and in this post I will explain why.
Connection Management
API Gateway
With API Gateway you need to manage your own connections. This means you need to listen to $connect and $disconnect events and store connection IDs in a database (DynamoDB or Redis would be very good choices). As they do not guarantee a $disconnect for every connection you are responsible for your own garbage collection on top of this.
IoT Core
IoT Core manages everything for you! Optionally you can still listen for connect and disconnect events using Lifecycle events.
Messaging
API Gateway
Sending a message is done through the API but allows to send to 1 connection at a time.
If you want to send a message to all connected users, you will need to loop over all connections in your database and send them a message individually. You could offload this work to an SQS queue + Lambda function but it's not ideal.
A noteable feature is data transformation which allows to transform inputs and outputs.
IoT Core
Because IoT Core is an MQTT broker, your clients can subscribe (including wildcards!) and publish to topics. Sending a message to all connected clients from the server is as easy as letting your clients subscribe to the same topic and then doing a single publish API request to the topic to send the message to everyone at once. You can also create a topic for an individual user by their user ID (eg: user/2d543007-2951-43cc-bec4-96f6649b4f53) and send individual messages if needed of course. Keep in mind however that depending on how you set up access control, others could potentially read along.
Integrations
API Gateway
Using custom routes you can create your own integrations based on the body of the received message. Any non-matched messages can be routed to a default route. It is possible to trigger a Lambda, call HTTP endpoints but also there is a possibility to call other AWS Services (especially useful in combination with data transformation).
IoT Core
Handing received messages can be done with Rules which uses an SQL-like syntax. Rule Actions specify what to do when a rule is hit and there are tons of out of the box integrations available. These integrations include Lambda, SQS, SNS & Kinesis to name a few.
Optionally you can send messages to a rule directly to save on messaging costs!
Security
API Gateway
Controlling access for websockets is similar to REST APIs. The option to use a Lambda authorizers and IAM authorization is very flexible.
Other useful security features include request validation and account level and route-level throttling. Using a custom authorizer you can also implement your own throttling mechanism.
IoT Core
Security in IoT Core has much more options than API Gateway since it has many different use cases. Luckily custom authentication allows for a lot of flexibility. This option doesn't only make it possible to allow or deny access to IoT Core but also specify exactly what is allowed to be done (ie: read only on specific topics). I will create a separate post about subject this soon.
There is also another option that doesn't seem to be documented; a pre-signed url! I found a library that does the trick perfectly. To control what the user can do you must create a IAM user for it with the right permissions on iot:Connect, iot:Subscribe, iot:Receive and iot:Publish and use that to sign the request. Your client can then simply connect to the pre-signed url and do it's thing. The pre-signed url is reusable for multiple users which means you can cache it on your CDN. Keep in mind this does give every user the same permissions but in some cases this is exactly what you need.
In all cases, whenever the client tries to do something it isn't allowed (ie: publish to a topic which the IAM user doesn't have permission to) the server will disconnect the client.
IoT Core does not seem to have any request validation or throttling features as of this moment.
Pricing
Pricing obviously is a bit different per region but overall the differences are similar. For this post I've chosen eu-west-1 since that is my region of choice.
API Gateway
- 1 Million connection minutes: $0.285
- Messages measured per: 32 KB
- First 1 billion: $1.14 per million
- Over 1 billion: $0.94 per million
IoT Core
- 1 Million connection minutes: $0.08
- Messages measured per: 5 KB
- Up to 1 billion messages: $1.00 per million
- Next 4 billion messages: $0.80 per million
- Over 5 billion messages: $0.70 per million
- Rules triggered: $0.15 per million
- Rule Actions executed: $0.15 per million
Messages sent or received using the reserved topics of Basic Ingest or Alexa Voice Service Integration for AWS IoT Core are free.
Key Differences
- IoT Core connection minutes are ~3 times cheaper than API Gateway
- Message pricing looks similar but API Gateway meters messages per 32 KB while IoT Core meters messages per 5 KB, a ~6.5 times difference
- Receiving and sending messages in IoT Core can be free in some cases
- IoT Core charges a fee for Rules and Rule actions, in API Gateway you will be charged for Lambda execution time instead.
Depending on how many messages, the size of your messages and your use case you can calculate easily which one would be cheaper to use. Don't forget though that with API Gateway you do need a database to manage your connections which also costs you extra. Also for sending messages in API Gateway you will most likely need a queue and a Lambda which adds more cost.
Scalability
I've selected some of the most important limits for both services.
API Gateway
New connections per second per account per region: 500
Concurrent connections: no enforced limit (!)
500 new connections per second is a lot and can be increased if needed. Having no limit is very cool but managing these connections in your own database might also add some challenges.
IoT Core
New connections per second per account: 500
Subscriptions per second per account: 500
Inbound publish requests per second per account: 20.000
Outbound publish requests per second per account: 20.000
Maximum concurrent client connections per account: 500.000
This seems to be a bit strict compared to API Gateway, luckily all these limits can be increased. I've had to do this a few times and I was able to set both inbound and outbound publish requests per second to 50.000 but it took some effort with the AWS support team to get it done (and it's understandable why).
I couldn't really find what happens if you exceed these publish limits. Does the message get sent to a limited amount of users, does no one receive it, will it get spread or queued automatically? If anyone has more information on this, please let me know about it as I would love to know more about it.
A possible solution could be to split users in multiple groups round robin and send a message to every group with some delays. On the client side, when sending messages, adding a random delay and doing a exponential back off retry mechanism can be very effective.
For most scenarios the default limits will probably be more than enough and you don't need to worry about these things at all.
Summary
Both are great services with their ups and downs. The main reasons why I prefer IoT Core over API Gateway:
- No connection management
- Working with topics instead of individual connections
- Much cheaper if you can keep the size under 5 KB per message
- Less code required
Thanks
Thanks to my colleague Iran Reyes who provided some good insights on the usage of API Gateway Websockets.
Top comments (0)