Introduction
Hey, DEV people! 😉 Today, I'll cover the topic of working with a Message broker called RabbitMQ in your Go projects. There will be both a theoretical part and practice.
Of course, the article is more aimed at those who just want to understand this topic. But I call on more experienced people to help cover this topic even better in the comments on this article. Together we can do a lot!
📝 Table of contents
- What is a message broker?
- What problems will be able to solve?
- Full project code
- Setting up RabbitMQ
- Setting up Fiber as a sender
- Setting up a message consumer
- Running the project
What is a message broker?
This is an architectural pattern in distributed systems, where a message broker is an application that converts a single protocol message from the source application to a protocol message from the destination application, thereby acting as an intermediary between them.
Also, the tasks of the message broker include:
- Checking the message for errors;
- Routing to specific receiver(s);
- Splitting the message into several smaller ones, and then aggregating the receivers' responses and sending the result to the source;
- Saving the messages to a database;
- Calling web services;
- Distributing messages to subscribers;
🤔 But what is it anyway? Well, let's translate it into our language.
If you simplify this huge description, you can portray the message broker as a post office in real life (which you have encountered many times):
- A sender (user of your product) brings a parcel (any data) to the post office and specifies the addressee for receipt (another service).
- A post office employee accepts the parcel and places it in the storage area (puts it in the queue to be sent) and issues a receipt that the parcel has been successfully accepted from the sender.
- After some time, the parcel is delivered to the addressee (another service), and he doesn't have to be at home to accept the parcel. In this case, his parcel will wait in a mailbox until he receives it.
What problems will be able to solve?
One of the most important problems that can be solved by using this architectural pattern is to parallelize tasks with a guaranteed result, even if the receiving service is unavailable at the time of sending the data.
With the total dominance of microservice architecture in most modern projects, this approach can maximize the performance and resilience of the entire system.
👌 It sounds a bit confusing... but let's use the post office analogy again!
Once the sender gives his parcel to the post office employee, he no longer cares how his parcel will be delivered, but he does know that it will be delivered anyway!
Full project code
For those who want to see the project in action:
koddr / tutorial-go-fiber-rabbitmq
📖 Tutorial: Working with RabbitMQ in Golang by examples.
Setting up RabbitMQ
As usual, let's create a new Docker Compose file:
# ./docker-compose.yml
version: "3.9"
services:
# Create service with RabbitMQ.
message-broker:
image: rabbitmq:3-management-alpine
container_name: message-broker
ports:
- 5672:5672 # for sender and consumer connections
- 15672:15672 # for serve RabbitMQ GUI
volumes:
- ${HOME}/dev-rabbitmq/data/:/var/lib/rabbitmq
- ${HOME}/dev-rabbitmq/log/:/var/log/rabbitmq
restart: always
networks:
- dev-network
networks:
# Create a new Docker network.
dev-network:
driver: bridge
☝️ Please note! For the initial introduction to RabbitMQ we will not create a cluster and use a load balancer. If you want to know about it, write a comment below.
Setting up Fiber as a sender
To connect to the message broker, we will use the Advanced Message Queuing Protocol or AMQP
for short. The standard port for RabbitMQ is 5672
.
Okay, let's write a simple data sender using Fiber web framework:
// ./sender/main.go
package main
import (
"log"
"os"
"github.com/gofiber/fiber/v2"
"github.com/gofiber/fiber/v2/middleware/logger"
"github.com/streadway/amqp"
)
func main() {
// Define RabbitMQ server URL.
amqpServerURL := os.Getenv("AMQP_SERVER_URL")
// Create a new RabbitMQ connection.
connectRabbitMQ, err := amqp.Dial(amqpServerURL)
if err != nil {
panic(err)
}
defer connectRabbitMQ.Close()
// Let's start by opening a channel to our RabbitMQ
// instance over the connection we have already
// established.
channelRabbitMQ, err := connectRabbitMQ.Channel()
if err != nil {
panic(err)
}
defer channelRabbitMQ.Close()
// With the instance and declare Queues that we can
// publish and subscribe to.
_, err = channelRabbitMQ.QueueDeclare(
"QueueService1", // queue name
true, // durable
false, // auto delete
false, // exclusive
false, // no wait
nil, // arguments
)
if err != nil {
panic(err)
}
// Create a new Fiber instance.
app := fiber.New()
// Add middleware.
app.Use(
logger.New(), // add simple logger
)
// Add route.
app.Get("/send", func(c *fiber.Ctx) error {
// Create a message to publish.
message := amqp.Publishing{
ContentType: "text/plain",
Body: []byte(c.Query("msg")),
}
// Attempt to publish a message to the queue.
if err := channelRabbitMQ.Publish(
"", // exchange
"QueueService1", // queue name
false, // mandatory
false, // immediate
message, // message to publish
); err != nil {
return err
}
return nil
})
// Start Fiber API server.
log.Fatal(app.Listen(":3000"))
}
As you can see, at the beginning we create a new connection to RabbitMQ and a channel to send data to the queue, called QueueService1
. With a GET request to localhost:3000/send, we can pass a needed text in a msg
query parameter, which will be sent to the queue and next to the consumer.
Now create a new Dockerfile called Dockerfile-sender
in the root of the project in which we describe the process of creating the container for sender:
# ./Dockerfile-sender
FROM golang:1.16-alpine AS builder
# Move to working directory (/build).
WORKDIR /build
# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download
# Copy the code into the container.
COPY ./sender/main.go .
# Set necessary environment variables needed
# for our image and build the sender.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o sender .
FROM scratch
# Copy binary and config files from /build
# to root folder of scratch container.
COPY --from=builder ["/build/sender", "/"]
# Command to run when starting the container.
ENTRYPOINT ["/sender"]
All that remains is to update the Docker Compose file so that it takes this Dockerfile into account when creating a container with RabbitMQ:
# ./docker-compose.yml
version: "3.9"
services:
# ...
# Create service with Fiber sender.
sender:
container_name: sender
ports:
- 3000:3000
build:
context: .
dockerfile: Dockerfile-sender
environment:
AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
restart: always
networks:
- dev-network
depends_on:
- message-broker
# ...
# ...
Setting up a message consumer
The message consumer should be able to accept messages from the broker's queue and output in the logs the message sent from the sender.
Let's implement such a consumer:
// ./consumer/main.go
package main
import (
"log"
"os"
"github.com/streadway/amqp"
)
func main() {
// Define RabbitMQ server URL.
amqpServerURL := os.Getenv("AMQP_SERVER_URL")
// Create a new RabbitMQ connection.
connectRabbitMQ, err := amqp.Dial(amqpServerURL)
if err != nil {
panic(err)
}
defer connectRabbitMQ.Close()
// Opening a channel to our RabbitMQ instance over
// the connection we have already established.
channelRabbitMQ, err := connectRabbitMQ.Channel()
if err != nil {
panic(err)
}
defer channelRabbitMQ.Close()
// Subscribing to QueueService1 for getting messages.
messages, err := channelRabbitMQ.Consume(
"QueueService1", // queue name
"", // consumer
true, // auto-ack
false, // exclusive
false, // no local
false, // no wait
nil, // arguments
)
if err != nil {
log.Println(err)
}
// Build a welcome message.
log.Println("Successfully connected to RabbitMQ")
log.Println("Waiting for messages")
// Make a channel to receive messages into infinite loop.
forever := make(chan bool)
go func() {
for message := range messages {
// For example, show received message in a console.
log.Printf(" > Received message: %s\n", message.Body)
}
}()
<-forever
}
Okay, in the same way with sender, let's create a new Dockerfile called Dockerfile-consumer
to describe the process of creating a container for the message consumer:
# ./Dockerfile-consumer
FROM golang:1.16-alpine AS builder
# Move to working directory (/build).
WORKDIR /build
# Copy and download dependency using go mod.
COPY go.mod go.sum ./
RUN go mod download
# Copy the code into the container.
COPY ./consumer/main.go .
# Set necessary environment variables needed
# for our image and build the consumer.
ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
RUN go build -ldflags="-s -w" -o consumer .
FROM scratch
# Copy binary and config files from /build
# to root folder of scratch container.
COPY --from=builder ["/build/consumer", "/"]
# Command to run when starting the container.
ENTRYPOINT ["/consumer"]
Yes, you guessed it! 😅 Once again, we will have to put a description in the Docker Compose file to create the container for the consumer:
# ./docker-compose.yml
version: "3.9"
services:
# ...
# ...
# Create service with message consumer.
consumer:
container_name: consumer
build:
context: .
dockerfile: Dockerfile-consumer
environment:
AMQP_SERVER_URL: amqp://guest:guest@message-broker:5672/
restart: always
networks:
- dev-network
depends_on:
- sender
- message-broker
# ...
Great! Finally, we're ready to put everything together and run the project.
Running the project
Just run this Docker Compose command:
docker-compose up --build
Wait about 1-2 minutes and make a few HTTP request to the API endpoint with different text in msg
query:
curl \
--request GET \
--url 'http://localhost:3000/send?msg=test'
Next, go to this address: http://localhost:15672
. Enter guest
both as login and password. You should see the RabbitMQ user interface, like this:
To simply view the logs and metrics inside the Docker containers, I recommend to use a command line utility, called ctop:
If you turn off the consumer container, but continue to make HTTP requests, you will see that the queue starts to accumulate messages. But as soon as the consumer is started up again, the queue will clear out because the consumer will get all the messages sent.
🎊 Congratulations, you have fully configured the message broker, sender, consumer and wrapped it all in isolated Docker containers!
Photos and videos by
- Vic Shóstak https://shostak.dev
- Fatos Bytyqi https://unsplash.com/photos/Agx5_TLsIf4
P.S.
If you want more articles (like this) on this blog, then post a comment below and subscribe to me. Thanks! 😻
❗️ You can support me on Boosty, both on a permanent and on a one-time basis. All proceeds from this way will go to support my OSS projects and will energize me to create new products and articles for the community.
And of course, you can help me make developers' lives even better! Just connect to one of my projects as a contributor. It's easy!
My main projects that need your help (and stars) 👇
- 🔥 gowebly: A next-generation CLI tool that makes it easy to create amazing web applications with Go on the backend, using htmx, hyperscript or Alpine.js and the most popular CSS frameworks on the frontend.
- ✨ create-go-app: Create a new production-ready project with Go backend, frontend and deploy automation by running one CLI command.
Top comments (4)
Great article ! Yes, would be interested to know about the cluster and LB with RabbitMQ.
Thanks, may be later.
I've been working on github.com/wagslane/go-rabbitmq for a few months now, would love it you checked it out!
Hi Im getting this error randomly on the sender container
UNEXPECTED_FRAME - expected content body, got non content body frame instead
Do you know hos to solve this? thank you
P.S. I had PHP 8 and Symfony 5 running a message sender, I had a 10ms response time, now I have 1ms with GO.
dev-to-uploads.s3.amazonaws.com/up...