SNS vs SQS? AWS Messaging Services - Know the difference

I noticed Helen released this good DEV.to post on SNS and SQS and I happen to have a video of the same nature lying around and thought I'd upload it to provide an explanation from a different perspective.

Article No Longer Available

A little bit more about Message Services

The 4 most important services you need to know when it comes to Application Integration are the following:

• SNS - PubSub (think pass along)
• SQS - Queueing (think batch)
• Kinesis - Real-time
• SWF - Queuing but with people

For the Solution Architect Associate and Certificated Cloud Practitioner you really need to know the difference between SNS and SQS.

MKS - Amazon Managed Streaming for Kafka

Most companies in BigData who want to use real-time messaging systems tend to use Kafka. Luckily AWS has a managed service called MKS (Managed Kafka Service).

Kinesis

The use case for Kinesis has been really good for gaming where you need a simple streaming queue. There is a Solutions Architect Professional exam question which asks how you would ingest millions of events from IoT game devices to then insert into a database such as DynamoDB to update a scoreboard, and so Kinesis is what you would keep in mind.

At ExamPro we built our own real-time event tracking (think GoogleAnalytics) so we can accurately track and improve how people learn on our platform.

SNS - Amazon Simple Notification Service

SNS is commonly used for sending simple internal emails. So for example when someone sign-ups on ExamPro I use the AWS SDK to send a message to an SNS topic which then sends me a plain text email.

Another way we use SNS is to trigger webhooks in your application. So for example, let us say you have a web-app which needs to extract data out of a PDF so you need to use a PDFTK which you have to run in a custom runtime in a Lambda function because of legacy reasons. When that lambda has processed that pdf you want to notify your web-app to tell the user it's done. You can have Lambda tell SNS which will then send an HTTP request to an api endpoint (webhook) to your web-app which will then update the user that the pdf has finished processing.

SQS - Amazon Simple Queue Service

When we send out an email such as confirmation or reset password on ExamPro we use SQS to decouple email sending from our app so it doesn't cause the app to hang. ExamPro is a Rails app and so this gem called Shoryuken makes it really easy to start working with SQS will little knowledge.

ruby-shoryuken / shoryuken

A super efficient Amazon SQS thread based message processor for Ruby

I'm looking for Shoryuken maintainers, are you interested on helping to maintain Shoryuken? Join our Slack

Shoryuken

Shoryuken sho-ryu-ken is a super-efficient Amazon SQS thread-based message processor.

Key features

• Rails Active Job
• Queue Load balancing
• Concurrency per queue
• Long Polling
• Batch processing
• Auto extend visibility timeout
• Exponential backoff
• Middleware support
• Amazon SQS CLI. See shoryuken help sqs

Requirements

Ruby 2.4 or greater.

Installation

Add this line to your application's Gemfile:

gem 'shoryuken'

If you are using AWS SDK version 3, please also add this line:

gem 'aws-sdk-sqs'

The extra gem aws-sdk-sqs is required in order to keep Shoryuken compatible with AWS SDK version 2 and 3.

And then execute:

$ bundle

Usage

Check the Getting Started page.

More Information

For more information check the wiki page.

Credits

Mike Perham, creator of Sidekiq, and everybody who contributed to it. Shoryuken wouldn't exist as it is without those…

View on GitHub

SWF

Nobody I know uses this service. It's used by AWS I believe to help coordinate a sequence of tasks with people and services which makes sense for Amazon when they need to have humans put things into boxes and then ship those boxes.

There isn't even a new AWS icon. I've never been able to figure out how to access this service. It shows up on exams.

Help me out make more free content

If you like this kind of AWS content the way you can help me out is adding me to Linked In and endorsing my skill for AWS. You should add me anyway I love connecting with people on DEV.to

Comments

[KP]

@exampro ok to ask a question on setting up SES Email notifications with SNS? I can't seem to get it to work.

[Andrew Brown 🇨🇦]

SNS to SES?

[KP]

Yes, what I'm trying to do is send an email through SES. And, subscribe to notifications on bounce / complaint / delivered emails to hit an endpoint. Can't seem to receive the pingbacks, so Iwas wondering if this is something you've dealt with

[Andrew Brown 🇨🇦]

So in SES you go to Domains > Click the Domain > Notifications and then you can set the SES notification.

Have you done this? I'm guessing you just want to get an SNS email on the bounce?

[KP]

@andrewbrown thank you for the response :)
I've tried this.

I am using SES to send email and am trying to get notified of bounces, complaints and successful deliveries by:

1. Email forwarding
2. SNS HTTP(S) POST notification (using this package to receive the notifications but I never get any: github.com/jdavidbakr/mail-tracker)

Help with either (but especially the HTTP SNS POSTs) would be great. Note that the ngrok url / tunnel in the examples below was functional, pasting the url for discussion sake...I can resurrect it if you need me to.

My settings:

1. From us-west-2.console.aws.amazon.com/s...
   Screenshot: prnt.sc/pquysj

2. SNS Topics and subscriptions:
   Topic: us-west-2.console.aws.amazon.com/s...
   Confirmed Subscription: us-west-2.console.aws.amazon.com/s...

3. Send email
   From: [e-mail] To: [e-mail]
   FAIL: No bounce email notification received (at either [e-mail] or [e-mail] )
   FAIL: No SNS HTTP(S) POST notification received at af84a2ab.ngrok.io/email/sns

4. Send email
   From: [e-mail] To: [e-mail]
   PASS: Complaint email notification received (at [e-mail] )
   FAIL: No SNS HTTP(S) POST notification received at af84a2ab.ngrok.io/email/sns

5. Send email
   From: [e-mail] To: [e-mail]
   FAIL: No "delivered" email notification received (at [e-mail] or [e-mail] )
   FAIL: No SNS HTTP(S) POST notification received at af84a2ab.ngrok.io/email/sns

So, I'm trying to get all the failed scenarios to pass, but especially the ones involving the SNS HTTP(S) POST notifications.

[Andrew Brown 🇨🇦]

Thank you for preparing that in good detail.

✔️So you have the SNS topics set correctly in SES
✔️So my next question was did you confirm your subscription which you did.
? The next thing I wasn't sure is what kind of subscription but since you are using ngrok and you mention HTTP(S) I am guessing you are trying to create an endpoint so that the SNS notification goes back into your laravel app so you can maybe store the bounces in your application's database.

So the question is where is the problem?
Is it SES, SNS or your Application?

So we need rule each one out, so now I ask if you have logging turned on about delivery status on the notification.

When this is turned on it will deliver that information to CloudWatch logs. If there are logs in there than we know it's your application, if you don't see anything in there then we know the problem is with SES.

If you already have this turned on its going to a bit tricky to debug and we'll have to think outside the box.

[KP]

@andrewbrown AWS has to be the most complex, non-user-friendly interface I have seen, thank you for your help. I truly appreciate it :)

You understood my situation perfectly...I am trying to get the SNS notification to my Laravel endpoint and log in to my database, or a file...
where end to end HTTP endpoint connectivity is established. Are you also using Laravel (or this package github.com/jdavidbakr/mail-tracker, by chance?)

Coming back to the problem...I did not have logging enabled previously, so I went ahead and did that (your screenshots really helped)

Went to the SNS console, clicked on the TOpic
Selected "Delivery Status Logging". Screenshot: res.cloudinary.com/practicaldev/im...
Clicked "Edit" and turn on Delivery Status Logging with HTTP(S): Screenshot: res.cloudinary.com/practicaldev/im...
Screenshot: prnt.sc/prdk3v

I then enabled Cloudwatch Logs for SMS messages:
docs.aws.amazon.com/sns/latest/dg/...

Also, before kicking off a test again, since my ngrok process (tunnel to my localhost so I can test SNS pings to my Laravel app) was dead, I restarted ngrok from the command line:

    ngrok http 8001

Btw, you can find my tentative ngrok tunnel (to my localhost:8001) at 1133c091.ngrok.io

[KP]

I got the new subscription confirmed
(was missing the full URL (the /email/sns part): 1133c091.ngrok.io/email/sns

Screenshot: prnt.sc/prdro5

and I now have logging enabled per your instructions...

[KP]

@andrewbrown

Another update: I found documentation pointing me to cloudwatch, so I see where the logs will flow into.

This is what I did next:
From: [e-mail] To: [e-mail]
PASS: Complaint email notification received (at [e-mail] )

This is what the cloudwatch logs show:
Screenshots:
prnt.sc/prdufv
prnt.sc/prdukf

{
    "notification": {
        "messageMD5Sum": "8f10e089c532ef6f078e9860cd028d24",
        "messageId": "70c867fc-4d68-5dc4-adae-31cabe42a76f",
        "topicArn": "arn:aws:sns:us-west-2:753513409872:KeenBrain_Amazon_SNS_for_SES_Emails_Topic",
        "timestamp": "2019-11-02 02:58:50.382"
    },
    "delivery": {
        "deliveryId": "88181b7b-6a95-5980-8fa6-4d4d990f3bab",
        "destination": "https://1133c091.ngrok.io/email/sns",
        "providerResponse": "Internal Server Error",
        "dwellTimeMs": 112754,
        "attempts": 3,
        "statusCode": 500
    },
    "status": "FAILURE"
}

Does this mean there's some kind of internal server error in my application?
This is weird because I'm using an out-of-the-box package: github.com/jdavidbakr/mail-tracker

which supposedly is working for other people.

As you can see, 1133c091.ngrok.io/email/sns is a POST endpoint. Any idea how I can debug further?

[Andrew Brown 🇨🇦]

You are right on both.

SNS was sending the logs to CloudWatch and you found them.

The error being reported is from your Laravel app. So it's not SNS or SES and it's not ngrok.
It's your Laravel application.

I rarely use Laravel buts its very similar to Ruby on Rails.
You could be getting an error for multiple reasons in your application:

• Expects authentication for an endpoint
• Routes are incorrectly set
• The error is occurring within your controller action due to misconfiguration

The problem doesn't appear to be with mail-tracker. I think the bounced email is working, an SNS notification is being triggered, but when SNS delivers the response to your endpoint its fails there.

[KP]

@andrewbrown

Thank you for the response....getting closer, thanks to you.

So, it's an issue with the application / package combination. Are you able to access 1133c091.ngrok.io (you should see "Email Sent" and 1133c091.ngrok.io/email/sns (you should see an error like "The GET method is not supported for this route. Supported methods: POST.")

The Laravel application is simply using the mail-tracker package (I've followed all the install and config instructions in the readme to a tee. I'm not saying that you should read that (you've been a great help)...but couldn't it potentially be a problem with the package itself? The SNS notification receipt /endpoint (which the package is responsible for and provides) is what is erroring out at 1133c091.ngrok.io/email/sns

By the way, I get the same error in cloudwatch logs if I send email to :
[e-mail]
or [e-mail]

However, I get no email receipts to my sending email, even though I have SES email forwarding enabled:
prnt.sc/pre15a
Is this something you've seen? I only seem to be getting emails of complaints, but not of successful deliveries or bounces.

[David J. Felix 🔮]

One thing to note is that I see a lot of people reach past SNS instead to SQS when they're looking for something "reliable", even when their consumer is going to be Lambda. I think this is generally a mistaken assumption for "serverless" pipelines, because a Lambda integration with SNS actually has capacity for retry and is an "at least once" reliable integration, just like SQS (you can also make SQS be exactly-once delivery, but it's not default).

9/10 times when a developer asks me to help them set up an SQS lambda integration, we end up settling on an SNS lambda integration instead.

I tend to think SQS is a bit outdated compared to SNS and Kinesis, but it definitely still has its place as an integration layer, especially with things that can't use Lambda extensively.

[Andrew Brown 🇨🇦]

I agree. For lambdas. I don't see the need to use SQS and we use SNS just as you describe.
Both SQS and Kinesis are "good enough" solutions but there are certainly better streaming and queuing services so its easy to outgrow for find these messaging systems limited.

[jesús gollonet]

Interesting. Up until reading this I would have considered myself one of that lot.

When I started with Lambda I defaulted to SNS because that seemed more naturally event-driven than polling, but then a few times I ended up missing some notifications because I had some edge cases on a Lambda and had to take it down for whatever reason, then changed my heuristic: If I'm integrating with a 3rd party service which provides a reliable SLA I can use SNS, otherwise I'll default to SQS for peace of mind.

I was not aware of retry / "at least once" delivery on SNS, but it makes total sense. I'll keep it in mind next time I have to do it.

[David J. Felix 🔮]

The retry is actually in lambda. All SNS + Lambda integrations will be resent until lambda receives the message, since this is an HTTP integration behind the scenes. You have to use caution once it gets to lambda to ensure you don't accidentally drop the message while working on the code.

Make sure you're not actually taking your lambda offline if you want to ensure delivery -- You'll miss any message that is sent before a lambda integration exists. If you're using Javascript, I recommend making your lambda handlers return promises rather than using callbacks and "rejecting" all failures that you want to retry, as they'll enter the retry policy for the lambda. Here's some info on retry: docs.aws.amazon.com/lambda/latest/...

We've found that they seem to retry at a steady pace with some amount of backoff.

You can also set a dead letter queue for messages that exceed the retry limit, which can be processed in a normal queue manner or by some manual process:

docs.aws.amazon.com/lambda/latest/...

We find a lot of reliability in our lambda + SNS + DLQ setups, to the point that SQS almost never gets used. We're not overly cautious about changing lambda code, but you want to ensure that you do reject on failures and you always keep the lambda SNS integration alive between code updates.

[jesús gollonet]

ah Lambda! that makes sense. thanks a lot for the detailed answer. Will keep that in mind for next time.