Message Queues: Basics of highly scalable asynchronous communication for apps

Tags

Top Highlights

• Broadcasting of messages by a producer to many Queues is termed Fan Out. It is fairly simple to implement compared to Fan In where Queues consume messages from multiple Queues.

• MQs can be parallel MQs or FIFO (First In First Out) MQs. FIFO is harder to implement as it needs to make sure consumer reads it in a specific order. This limits the amount of messages allowed per second.

• When to use MQs MQs help with decoupling of services and apps that need to exchange data. If message producer and consumer doesn’t need to bind by an agreement and also be able to make changes independent of each other MQs are the ideal solution to communicate.

• Producers put messages into a topic (imagine a header on a notice board). Most MQ products available today lets you create a topic. Consumers subscribe to that topic regardless of the origin.

• Note that subscribers are difficult to implement as it needs exception handling mechanisms to handle ill-formatted messages, malicious payload inside messages, Dead Letter Queues etc. DLQ is an exception handling Queue where messages that failed to process can be moved and be hooked to an alarm so we can investigate.

• A popular software development pattern for implementing message passing is called publish/subscribe or pub/sub. It is implemented as the subscriber polling the Queue usually through an HTTP request or through webhooks.

• Apart from SNS-SQS combo, AWS offers Eventbridge which introduces the concept of an event bus with no separation of topics and Q’s. All messages are of a structured format and all of them pass through Eventbridge. A topic is embedded inside the message to channel them to subscribers. This unified pattern helps define more rules on the event bus and allows advanced features on the bus like archival, monitoring, alerting, replay etc.

• Perhaps the more popular tools are Kafka and RabbitMQ. Kafka messages are called events. Events can be published into a topic. A topic is partitioned using a key. Each partition is ordered in FIFO. Events can be streamed through sockets or polled by consumers and is widely used for event streaming applications like IoT and web analytics. Number of partitions (servers) determine how much Kafka can distribute load. Kafka is known to be difficult to install and maintain, and its managed services are expensive than SQS/SNS.

• RabbitMQ is an open source tool like Kafka. It has direct support for Queues and messages. Queues are FIFO by default. This is a much easier tool to install and maintain.

• 1. Producers publish messages to a topic. 2. Have a separate Queue for each receiver 3. Link each Queue to the topic that its receiver is interested in.

• Synchronous: Make an HTTP request through standard HTTP operations like post, put, get etc. Asynchronous: Place your request on a message queue.

• Grady Booch, the much respected software architect asserts that “at some level of abstraction all complex systems are message passing systems". He means all systems, from cell biology to plate tectonics, not just software systems.

• The ideal design pattern for MQs go like this: