Amazon’s Simple Queue Service (SQS) provides durable messaging guarantees and is an excellent backbone for messaging services. However, SQS does not support “fan-out” of messages so that multiple consuming services can each receive a copy of a message. This means that true publish-subscribe messaging requires some additional work. This post describes some architectural choices that provide durable publish-subscribe messaging using SQS by tracking messaging subscribers using a database, and matching published messages to interested subscribers.
Terminology in this post follows that of Enterprise Integration Patterns.
Terminology
Message Channel
A connection between two applications uses a Message Channel, where one application writes information to the channel and the other one reads that information from the channel.
Point-to-Point Channel
A Point-to-Point Channel ensures that only one receiver consumes any given message. If the channel has multiple receivers, only one of them can successfully consume a particular message. If multiple receivers try to consume a single message, the channel ensures that only one of them succeeds, so the receivers do not have to coordinate with each other. The channel can still have multiple receivers to consume multiple messages concurrently, but only a single receiver consumes any one message.
Point-to-Point Channels do not necessarily imply in order delivery of messages.
Publish-Subscribe Channel
A Publish-Subscribe Channel has one input channel that splits into multiple output channels, one for each subscriber. When an event is published into the channel, the Publish-Subscribe Channel delivers a copy of the message to each of the output channels. Each output channel has only one subscriber, which is only allowed to consume a message once. In this way, each subscriber only gets the message once and consumed copies disappear from their channels.
Recipient List
A recipient list allows for the creation of a channel (either publish-subscribe or point-to-point) for each message consumer. The recipient list inspects each incoming message, and determines the list of recipients that have registered interest in the message topic. The list then forwards the message to all required channels for delivery to consumers.
Recipient lists allow for multiple channels to be defined for a single message. Each message consumer defines their own channel for receiving messages, and can choose to subscribe using either a Point-to-Point or a Publish-Subscribe Channel.
Competing Consumers
Competing consumers are multiple consumers that all receive messages from a single point-to-point channel. When the channel delivers a message, any one of the consumers could potentially receive it. The messaging system’s implementation determines which consumer actually receives the message, but in effect the consumers compete with each other to be the receiver. Once a consumer receives a message, it can delegate to the rest of its application to help process the message.
This solution only works with Point-to-Point Channels; multiple consumers on a Publish-Subscribe Channel just create more copies of each message. The default implementation of Amazon SQS provides competing consumer semantics.
Conceptual Design
Arrows in this diagram indicate message flow, this flow can be achieved with either push-based or pull-based implementations at each stage.
In the preceding figure, the Durable Messaging System (DMS) provides the APIs necessary for clients to publish messages. Upon receipt of a message, the topic of the message is read to derive a list of recipients for this particular message. Each recipient corresponds to a Message Channel, which could be either a Point-to-Point or Publish-Subscribe Channel. The Channels are responsible for delivery of messages to consumers.
Implementation
We can implement the conceptual design using Amazon SQS as the Message Channel implementation and use SQL to manage the list of message recipients and their corresponding SQS queues.
Managing the Recipient List
To provide durable subscriptions requires having knowledge of who is subscribed to each queue. This requires explicit registration of subscription interest for each subscriber. Knowing this information, the durable messaging system is able to manage the recipient list and deliver published messages to the correct queue.
When a consumer registers to listen to a message topic, the durable messaging system creates a corresponding SQS queue to match to that consumer. This matching between consumer and queue is stored in SQL.
Amazon SQS as Channel Implementation
We leverage Amazon SQS as the Channel implementation that guarantees message delivery. Once the SQS queue confirms receipt of a message, SQS implements our requirements for guaranteed delivery, at-least-once delivery, competing consumers, and FIFO, without further implementation work. The job of the messaging system is then to connect published messages to the correct queue for delivery to consumers.
Published messages are routed to the correct SQS queues for final delivery to consumers by looking up the correct consumers and queues from the recipient list managed in SQL. Since SQS only delivers a single message to a single consumer, it does not support a publish-subscribe pattern where each message can be delivered to multiple subscribers. To support this feature, recipient list is used to fan-out incoming messages to multiple interested SQS queues.
Fan-out using Amazon SNS
An alternative implementation could use Amazon SNS to fan-out messages to multiple subscribers. The messaging system would then manage the mapping between SNS notifications and SQS queues. With such an implementation, you may be able to get rid of the SQL database that stores message subscription data. We chose to keep the SQL database so that we have greater control over subscribers and topics in the system.
Conclusion
By leveraging Amazon SQS we’ve designed a scalable messaging system that provides durable messaging guarantees with publish-subscribe semantics. This design is easily scalable and easy to maintain. This post should serve as a jumping-off point for what to consider when building a publish-subscribe messaging system using AWS.