What Is RabbitMQ? | Intro To Message Queue Tech | Seventh State

What is RabbitMQ? An Intro to Its Features & Common Use Cases

Why Rabbit? What is RabbitMQ? Is it suitable for applications? If so, how can it improve them? Why would I want to learn more about it? These are questions that I asked when I was first introduced to RabbitMQ.
I’m Gabor, and I’m a RabbitMQ engineer and consultant. In my time working with RabbitMQ, I’ve learned that even experienced customers ask these questions.

What is Rabbit MQ used for?

RabbitMQ supports decoupling communication between services, making it easier to scale and maintain distributed systems.

Problem: Direct communication between services can be tightly coupled, rigid, error-prone, and harder to scale or maintain. These challenges can lead to inefficiency, bottlenecks, and even service disruptions.

Solution: A mediator (broker software), providing queuing, ensuring message delivery, and balancing the load between services.

For example, using a HTTP API requires a service to know the specific address in order to communicate. Any changes made to one service can break functionality in all other dependent services, requiring coordinated updates. For high-throughput applications, HTTP can also result in a significant amount of latency due to network overhead, handshakes, and retries.

RabbitMQ offers a solution to these issues by acting as a mediator between services. It eliminates congestion concerns by queuing messages until they’re ready to be processed. It can ensure reliable message delivery, and distribute the workload evenly among services. 

It also removes dependence on specific addresses. The consumer system only needs to know the queue name(s) it’s consuming from, meaning that the system publishing messages can be reconfigured without disruption.

What Problem Does RabbitMQ Message Broker Solve?

Before we delve into what RabbitMQ is and how to use it, it is worth learning more about the problem domain itself. Communication between different RabbitMQ services (a.k.a computers) is an age-old problem.

On one hand, there are the different messaging protocols defining the means of sending and receiving messages and the properties of the communication. Some examples of such protocols include SMTP, FTP, HTTP or WebSockets (to name a few), which are all based on TCP/UDP. They deal with the formatting, reliability and finding the correct recipient of a message.

On the other hand, we can explore the communication from the perspective of the message. It exists in one system, then it is transported to another, gets transformed, that is, the message has a lifecycle. As it travels from one messaging system to another, we should be aware of where the message is, and who owns it at any given point of time.

The communication protocols mentioned above can make sure that the ownership (and the “physical” location) of the message is transferred from one system to the other (although it may take some time to execute this transaction). We can consider the transfer to be a transaction between the two parties while both are present. Most of the time, this active exchange is desirable, e.g. asking for the status of the service and expecting a timely and accurate answer.

An example from the physical world would be calling somebody over the phone: 

1) We start the call

2) Wait for the other party to answer

3) have a nice discussion

4) hang up the phone

But there are other times when we don’t need the answer, we just need the receiver to take ownership of the message and do its job. In this case, we need an intermediary agent, another system to take ownership of the message (temporarily) and make sure that the message reaches its destination. To push the phone example further, the other party is not available at the moment, so we leave a voice message. The voicemail service will notify the intended receiver.

This asynchronous (delayed) message delivery is what RabbitMQ provides. Obviously, it can do more than a simple answering machine, so let’s explore some of the options it provides below:

(If you are interested in learning more about the history of this open-source message broker software, I recommend the first chapter of “RabbitMQ in Action” by Alvaro Videla and Jason Williams. It will reveal the answer to why it is named after Rabbits).

What is RabbitMQ? And How Does RabbitMQ Work?

RabbitMQ is a free, open-source, extensible, and reliable message queuing solution. It is a powerful message broker that implements the Advanced Message Queuing Protocol (AMQP), but is also able to be used with other popular messaging solutions like MQTT. It's known for its high availability, scalability, reliability and fault tolerance. RabbitMQ is built in Erlang OTP programming language, a technology tailored for building stable, reliable, fault tolerant and highly scalable systems that possess native capabilities of handling very large numbers of concurrent operations, like RabbitMQ and other systems like WhatsApp, MongooseIM, to mention a few.
At a very high level, it is a middleware layer that enables different services in your application to communicate with each other without worrying about message loss while providing different quality of service (QoS) requirements. RabbitMQ also supports fine-grained and efficient message routing enabling extensive decoupling of applications.

Common Use Cases for RabbitMQ

RabbitMQ is often used in scenarios like breaking down monoliths into microservices architecture, stream processing, and as an intermediary between applications and legacy systems. 
Below are examples of RabbitMQ use cases:

Using RabbitMQ as a General Message-Bus

When a monolith system is broken down to separate subsystems, one of the biggest problems that needs solving is which communication technology to use. A solution like Mulesoft, or MassTransit can “wire” services by declaring HTTP listeners and senders. This kind of solution treats RabbitMQ as a black box, but is still able to leverage the capabilities of RabbitMQ. As an example of direct communication, let’s use HTTP to “connect” the individual services. While it is a well-supported and solid choice, it has some drawbacks:
1) The discovery of services is not solved: A possible solution is to use DNS. As the system scales and grows, so too does the complexity of finding and balancing this load. RabbitMQ can mitigate the increased complexity of the solution.
2) The communication is ephemeral: Messages are prone to being dropped or duplicated on the network layer. If a service is unavailable temporarily, the delivery fails.
RabbitMQ can help in both cases by utilising message queues as a means of transport. Services can publish and consume messages, which decouples the end-to-end message delivery from the availability of the destination service. If a consuming service is temporarily unavailable, unlike HTTP, the message is safely buffered and retained in RabbitMQ, and eventually delivered when the service comes back online.

Discoverability is simplified too. All we need to know is where RabbitMQ is and what the queue name is. Although it seems like this just reinvents the problem, this is scalable. The queue name acts as the address of the service. Consuming messages from the queues by the individual services offer a means for scalability, i.e. each message queue can serve multiple consumers and balance the load. There is no need to change the message queue configuration already built into the services.
This moderately static message queue configuration pushes RabbitMQ to a middleware layer where a solid design can guarantee a stable service quality in the long term.

RabbitMQ as an Advanced Routing Layer for Micro-Services

On the other end of the spectrum is an architecture which is more fluid and adapts to the ever-changing needs of many micro-services. What makes it shine in this environment is the very powerful routing capabilities RabbitMQ supports.
 The routing logic is implemented in (so-called) different exchange types that can be dynamically created by the application when needed. The destination services create the queues that they wish to consume from, then bind them to exchanges by specifying a pattern for the keys the publishers can use when publishing the message. (Think about these keys as metadata that the exchanges can use to route and ensure the delivery of messages to one or more queues.)

RabbitMQ comes with four useful exchange types that cover most of the use cases for messaging:

1) Direct exchange

This will deliver the incoming message to any queue whose binding key exactly matches the routing key of the message. If you bind the queues with the queue name as routing keys, then you can think about it as a one-to-one message delivery. It is simple to deliver the same message to multiple queues by using the binding keys for multiple queues.

2) Topic exchange

This will deliver the incoming message to any queue whose wild-card binding key matches the routing key of the published message. Binding keys can contain wild-card matching criteria for a compound routing key. (e.g. the binding key logs.*.error will match the routing keys logs.accounting.error and logs.ui.error). This enables us to write simple services where the logic is well contained, and the message will arrive to the correct services through the “magic” of RabbitMQ.

3) Fanout exchange

Some messages need to be delivered to all queues, this is where a fanout exchange can be used instead of writing an elaborate multicast logic in the application. With a RabbitMQ fanout exchange, each service binds the appropriate queue to the exchange without need to specify a binding key, and it all happens automatically. If a binding key is specified, the fanout exchange will simply ignore it and still route/broadcast messages to all queues bound to it.

4) Headers exchange

This exchange leverages the structure of AMQP messages and is capable of complex routing based on the headers (including custom ones) of the AMQP message. Headers are metadata attached to each message sent via AMQP.
In addition to exchanges, there are other useful features of RabbitMQ that enable the implementation of very complex messaging logic. Some of the most important features include:

1) Custom plug-ins

RabbitMQ allows its users to add plug-ins, and is therefore highly extensible, which enhances its functionality beyond that of a basic message broker like Kafka. Almost every aspect of RabbitMQ is customisable, including the management, authentication and authorisation, back-up solutions, and clustering.

2) Clustering

When a single RabbitMQ server is not enough, multiple RabbitMQ brokers can be connected to work together and scale the system. It can enable RabbitMQ to process more messages or increase resilience to errors.

3) Quality of Service tuning

Time-sensitive message delivery can be helped by attaching a TTL (Time-to-Live) value to either the message or the queue. Timed out messages can be automatically delivered to a Dead-letter queue. Combining ordinary routing logic and these extra features can lead to highly advanced routing logics. Another useful feature is message prioritisation, or priority queues, where the publisher can assign a priority level to each message. It is also possible to limit the number of unacknowledged messages, which allows for the performance tuning of the consuming services, in this case, RabbitMQ applies a back-pressure mechanism.

RabbitMQ Integrated Into Legacy Systems

In the previous use-case, I mentioned the possibility of using plug-ins to extend the functionality of RabbitMQ. This powerful feature allows RabbitMQ to act as a mediation layer between your RabbitMQ native (AMQP capable) services and other legacy applications. Some notable examples include:
1) Using RabbitMQ as an MQTT broker by simply enabling a plug-in. This opens up the landscape to many IoT technologies.
2) RabbitMQ’s JMS (Java Message Service) plug-in, which allows RabbitMQ to communicate with any JMS capable messaging solution.
3) If your application is using a proprietary protocol for communicating, it is possible to develop a custom plugin to connect to any such services.

Need Help Setting up RabbitMQ Message Broker?

As the above use cases demonstrate, there is hardly anything that this open-source message broker can’t communicate with. But as with anything in life, it has a price (check our support packages and service offerings). Although configuring and running RabbitMQ is mostly straightforward, sometimes the mere number of features RabbitMQ offers can be overwhelming.