Preserving Message Order with Apache ActiveMQ


There are some use cases and business scenarios where you have to process messages in order. Basically, when multiple consumers receive messages from a queue and process them in parallel, message order is not preserved.

Apache ActiveMQ has the capability to process messages in order using a feature called Exclusive Consumers. With Exclusive Consumers all messages from a single queue are processed by only one consumer (other consumers listening on the same queue will take over if the exclusive consumer fails).

So, fundamentally the diagram below shows how Exclusive Consumers work where Consumer A and Consumer B are both listening to the same queue but only the Consumer A is consuming messages from it.






The Exclusive Consumer feature is configured at the consumer level, so at the code level you have to explicitly say what queue you're listening in and also say you're an an exclusive consumer.

Here is a piece of code showing how to set the Exclusive Consumer:


queue = new ActiveMQQueue(“TEST.QUEUE?consumer.exclusive=true”);


That's very simple and still powerful…


But, there's more to that! You also have the choice to set a priority for each consumer.

One way to ensure that high-priority applications get messages to process is to set the consumer priority. When multiple consumer request exclusive access to a queue, the consumer with the highest priority is selected by the broker.

So, another case for Exclusive Consumer priority is the machine speed you have available. You may want to give a higher priority for consumers deployed on faster machines and set the priority according to the resources you have available.

To set the Exclusive Consumer priority all you have to do is set another option on the consumer code like the below configuration:


queue = new ActiveMQQueue(“TEST.QUEUE?consumer.exclusive=true&consumer.priority=10");

The diagram below shows how the consumer priority would play with these settings.






One disadvantage of Exclusive Consumers is that you have active consumers doing nothing but we are going to cover how ActiveMQ helps you avoid that situation on a future post.

Stay tuned!


Comments

  1. Replies
    1. Thanks omkar. I'm glad you found this post useful.

      Delete
  2. if the consumers are the same priority, it seems to be sent in a round robin manner. How can you simulate a case where the message order is not received by the consumer's broker in the same sequence that the producer's broker received the data? My thought was to use ActiveMQ broker interceptors. Any thoughts on how to use the BrokerFilter public or protected members to simulate messages being funneled in a sequence that wasn't the original sequence? For example, having an interceptor which is derived from BrokerFilter and "holding onto message #4" until message #5 comes through and then releasing #4 to be consumed by the consumer.

    ReplyDelete
  3. Hi Marcelo, can this be configured on the queue level instead of consumer level?

    ReplyDelete
    Replies
    1. Hi Ahmed, Assuming you are referring to some sort of broker level configuration, you can use per destination policies. That topic is discussed here: http://activemq.apache.org/how-can-i-support-priority-queues.html . That should give you a good idea on how to do it.

      Delete
  4. Thanks Marcelo, in fact I don't need to set priorities for the message, what I need is making my consumers exclusive to achieve active/passive model, only one consumer is active and the other remains idle and works only when the master consumer is down, I don't want the two nodes working together because order should be preserved.

    My problem is that I am using Mule and I am not sure if Mule ActiveMQ connector has support for Exclusive Consumer option, does it?

    ReplyDelete
    Replies
    1. Check the Mule ActiveMQ connector documentation for more information.

      Delete
    2. Check the Mule ActiveMQ connector documentation for more information.

      Delete
  5. hey I want to do using MDB can you help me

    ReplyDelete
  6. hi could you tell me how to do using mdbs

    ReplyDelete
  7. Hi Jabali, I got exact same request to implement in my application but its not working expectedly. I have posted this in Stack Overflow:
    http://stackoverflow.com/questions/37001731/activemq-queue-priority-consumer-mechanism-not-working-expectedly

    Hope to hear from you soon.

    ReplyDelete
  8. And Can you please elaborate difference between using exclusive=true and setting priority? As both of these are intended to make sure in order delivery of messages.

    Why we need to specify priority along with consumer when we mark it as exclusive true? And will we have to mark each consumer exclusive true?

    ReplyDelete

Post a Comment

Popular posts from this blog

Calling Web Services with Apache Camel

How to Declare Variables in MS-SQL Server Management Studio

Using HTTP-based endpoints with Apache Camel