Introduction to RabbitMQ
RabbitMQ is a message-queueing software also known as a message broker or queue manager. Simply said; it is software where queues are defined, to which applications connect in order to transfer a message or messages.
A message can include any kind of information. It could, for example, have information about a process or task that should start on another application (which could even be on another server), or it could be just a simple text message. The queue-manager software stores the messages until a receiving application connects and takes a message off the queue. The receiving application then processes the message.
A message broker acts as a middleman for various services (e.g. a web application, as in this example). They can be used to reduce loads and delivery times of web application servers by delegating tasks that would normally take up a lot of time or resources to a third party that has no other job.
In this guide, we follow a scenario where a web application allows users to upload information to a website. The site will handle this information, generate a PDF, and email it back to the user. Handling the information, generating the PDF, and sending the email will, in this example case, take several seconds. That is one of the reasons why a message queue will be used to perform the task.
When the user has entered user information into the web interface, the web application will create a “PDF processing” message that includes all of the important information the user needs into a message and place it onto a queue defined in RabbitMQ.
The basic architecture of a message queue is simple – there are client applications called producers that create messages and deliver them to the broker (the message queue). Other applications, called consumers, connect to the queue and subscribe to the messages to be processed. Software may act as a producer, or consumer, or both a consumer and a producer of messages. Messages placed onto the queue are stored until the consumer retrieves them.
When and why should you use RabbitMQ?
Message queueing allows web servers to respond to requests quickly instead of being forced to perform resource-heavy procedures on the spot that may delay response time. Message queueing is also good when you want to distribute a message to multiple consumers or to balance loads between workers.
The consumer takes a message off the queue and starts processing the PDF. At the same time, the producer is queueing up new messages. The consumer can be on a totally different server than the producer or they can be located on the same server. The request can be created in one programming language and handled in another programming language. The point is, the two applications will only communicate through the messages they are sending to each other, which means the sender and receiver have low coupling.
- The user sends a PDF creation request to the web application.
- The web application (the producer) sends a message to RabbitMQ that includes data from the request such as name and email.
- An exchange accepts the messages from the producer and routes them to correct message queues for PDF creation.
- The PDF processing worker (the consumer) receives the task message and starts processing the PDF.
Messages are not published directly to a queue; instead, the producer sends messages to an exchange. An exchange is responsible for routing the messages to different queues with the help of bindings and routing keys. A binding is a link between a queue and an exchange.
Message flow in RabbitMQ
- The producer publishes a message to an exchange. When creating an exchange, the type must be specified. This topic will be covered later on.
- The exchange receives the message and is now responsible for routing the message. The exchange takes different message attributes into account, such as the routing key, depending on the exchange type.
- Bindings must be created from the exchange to queues. In this case, there are two bindings to two different queues from the exchange. The exchange routes the message into the queues depending on message attributes.
- The messages stay in the queue until they are handled by a consumer
- The consumer handles the message.
TYPES OF EXCHANGES
- Direct: The message is routed to the queues whose binding key exactly matches the routing key of the message. For example, if the queue is bound to the exchange with the binding key pdfprocess, a message published to the exchange with a routing key pdfprocess is routed to that queue.
- Fanout: A fanout exchange routes messages to all of the queues bound to it.
- Topic: The topic exchange does a wildcard match between the routing key and the routing pattern specified in the binding.
- Headers: Headers exchanges use the message header attributes for routing.
RABBITMQ AND SERVER CONCEPTS
Some important concepts need to be described before we dig deeper into RabbitMQ. The default virtual host, the default user, and the default permissions are used in the examples, so let’s go over the elements and concepts:
- Producer: Application that sends the messages.
- Consumer: Application that receives the messages.
- Queue: Buffer that stores messages.
- Message: Information that is sent from the producer to a consumer through RabbitMQ.
- Connection: A TCP connection between your application and the RabbitMQ broker.
- Channel: A virtual connection inside a connection. When publishing or consuming messages from a queue – it’s all done over a channel.
- Exchange: Receives messages from producers and pushes them to queues depending on rules defined by the exchange type. To receive messages, a queue needs to be bound to at least one exchange.
- Binding: A binding is a link between a queue and an exchange.
- Routing key: A key that the exchange looks at to decide how to route the message to queues. Think of the routing key like an address for the message.
- AMQP: Advanced Message Queuing Protocol is the protocol used by RabbitMQ for messaging.
- Users: It is possible to connect to RabbitMQ with a given username and password. Every user can be assigned permissions such as rights to read, write and configure privileges within the instance.
At the beginning of this article series, we had one producer (the website application) and one consumer (the PDF processing application). If the PDF processing application crashes, or if many PDF requests are coming at the same time, messages would continue to stack up in the queue until the consumer starts again. It would then process all the messages, one by one.
Set up a RabbitMQ instance
To be able to follow this guide you need to set up a CloudAMQP instance or download and install RabbitMQ. CloudAMQP is a hosted RabbitMQ solution, meaning that all you need to do is sign up for an account and create an instance. You do not need to set up and install RabbitMQ or care about cluster handling, CloudAMQP will do that for you. CloudAMQP can be used for free with the plan little lemur. Go to the plan page and sign up for any plan and create an instance.
When your instance is created, click on details for your instance to find your username, password, and connection URL for your cloud-hosted RabbitMQ instance.
Install RabbitMQ on Ubuntu
Since RabbitMQ is written in Erlang, you need to install Erlang before you can use RabbitMQ:
sudo dpkg -i esl-erlang_20.1-1\~ubuntu\~xenial_amd64.deb
Enable RabbitMQ PPA repository on your system. Also, import rabbitmq signing key on your system. Use the following commands to do this.
- echo ‘deb http://www.rabbitmq.com/debian/ testing main’ | sudo tee /etc/apt/sources.list.d/rabbitmq.list
- wget -O- https://www.rabbitmq.com/rabbitmq-release-signing-key.asc | sudo apt-key add –
After that update apt cache and install RabbitMQ server on your system.
- sudo apt-get update
- sudo apt-get install rabbitmq-server
Manage RabbitMQ Service
After completing installations, enable the RabbitMQ service on your system. Also, start the RabbitMQ service. Use one of the below methods sysvinit for older systems or systemctl for the latest operating system.
Using Init –
sudo update-rc.d rabbitmq-server defaults
sudo service rabbitmq-server start
sudo service rabbitmq-server stop
Using Systemctl –
sudo systemctl enable rabbitmq-server
sudo systemctl start rabbitmq-server
sudo systemctl stop rabbitmq-server
Create Admin User in RabbitMQ
By default rabbitmq creates a user named “guest” with password “guest”. You can also create your own administrator account on RabbitMQ server using following commands. Change password with your own password.
sudo rabbitmqctl add_user admin password
sudo rabbitmqctl set_user_tags admin administrator
sudo rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
Setup RabbitMQ Web Management Console
RabbitMQ also provides and web management console for managing the entire RabbitMQ. To enable web management console run following command on your system. The web management console helps you with managing the RabbitMQ server.
sudo rabbitmq-plugins enable rabbitmq_management
RabbitMQ dashboard starts on port 15672. Access your server on the port to get the dashboard. Use the username and password created in previous step.
After login, you will get the RabbitMQ management web interface dashboard.
THE MANAGEMENT INTERFACE – MANAGEMENT AND MONITORING
RabbitMQ provides a web UI for the management and monitoring of your RabbitMQ server. The RabbitMQ management interface is enabled by default in CloudAMQP and a link can be found on the details page for your CloudAMQP instance.
From the management interface, it is possible to handle, create, delete and list queues. It is also possible to monitor queue length, check message rate, or change and add users permissions and much more.
PUBLISH AND SUBSCRIBE MESSAGES
RabbitMQ uses a protocol called AMQP by default. To be able to communicate with RabbitMQ you need a library that understands the same protocol as RabbitMQ. Download the client library for the programming language that you intend to use for your applications. A client library is an application programming interface (API) for use in writing client applications. A client library has several methods; in this case, to communicate with RabbitMQ. The methods should be used when you connect to the RabbitMQ broker (using the given parameters, hostname, port number, etc.), for example, or when you declare a queue or an exchange. There is a choice of libraries for almost every programming language.
Steps to follow when setting up a connection and publishing a message/consuming a message:
- Set up/create a connection object. The username, password, connection URL, port, etc., will need to be specified. A TCP connection will be set up between the application and RabbitMQ when the start method is called.
- Create a channel in the TCP connection, then the connection interface can be used to open a channel through which to send and receive messages.
- Declare/create a queue. Declaring a queue will cause it to be created if it does not already exist. All queues need to be declared before they can be used.
- Set up exchanges and bind a queue to an exchange in subscriber/consumer. All exchanges must be declared before they can be used. An exchange accepts messages from a producer application and routes them to message queues. For messages to be routed to queues, queues must be bound to an exchange.
- In publisher: Publish a message to an exchange
In subscriber/consumer: Consume a message from a queue.
- Close the channel and the connection.
Using .Net Core With RabbitMQ For Async Operations
To create sample applications of how to use RabbitMQ in .net core please open following link –
Clients Libraries and Developer Tools
To get details of all supporting operating systems and client libraries, please open following link –