RabbitMQ first reactions

As promised this is the first note about stuff we are working on our current project. Based on the nature of our system an asynchronous type of solution is needed. Having worked previously with some JMS solutions for example SonicMQ the words Messages and Queues was stuck on my mind right away when the project began. 

OK, then if you remember my previous post I mentioned this is a "start-up" like project so we have to keep it cheap and OSS is the way to go. As consequence my first impression was hey! HornetQ its SUPER fast! 8.2Million messages per second. Lets try that! 

So my colleague starts doing the first tests and after a while we begin realizing that its maybe too big of a rocket to shoot a fly and not so friendly to start with. Meanwhile I immerse myself in pages blogs and forums and the term Advanced Message Queuing Protocol (AMQP) pokes on my eyes together with RabbitMQ which looks like the cool new kid on the messaging industry. So I do a few tests and with a very little configuration and coding I have a Hello World app publishing and reading messages. So again I say hey lets try that! I know my colleague hates me for changing my mind but I hope he forgave me after seeing how easy it was to set up and how it solved some problems we had.

So yes it RabbitMQ looks like a very capable and very flexible messaging solution at the moment where you can publish and subscribe to queues on the go. Create exchanges, topics, routing and many integration patterns without adding too much complication. 

Eventually we found the first problem. Priority Levels. Despite being part of the AMQP specification and being defined on the Java API the current version doesn't support it. On the official page they have this table with their compatibility mapping to the AMQP specification and sadly I find that indeed they don't have it YET implemented.

This is the compatibility for priorities on RabbitMQ for AMQP 0-9-1:

plannedMUSTbasic / 06
The server MUST implement at least 2 priority levels for basic messages, where priorities 0-4 and 5-9 are treated as two distinct levels.
doesn'tMAYbasic / 07
The server MAY implement up to 10 priority levels.

So hopefully soon enough a new release will come out with at least  the 2 priority levels management implemented. Hurry up guys!

Meanwhile you can use this post by Doug Barth as a reference on how they developed a workaround to this lack of functionality by RabbitMQ. We will try to develop a variant of this method of prioritizing messages and I'll let you know our results. It basically consists on two queues a top priority queue and a regular queue from which we will be polling the messages. Its not an actual priority implementation but its an approximate. that in our case could solve initial problem.

Would be nice to have some feedback on how would you work this around feel free to comment.

P.S: This is an obligatory book if you are building asynchronous messaging systems.

Comments

  1. Ana9:34 AM

    Well, isn't a bad idea having a couple of queues for solving the priority problem, and more so, if I undertstand right, that RabbitQ considers 2 priorities (in the future? that part I didn't get).

    Another thing you have to consider is that if you aren't using the "rocket" solution, even knowing that in the future you may need it, maybe you should consider having a queue for each kind of transaction or task that need it. You have to think in the future, it would be very difficult to change solutions later on.

    Good luck!!!

    ReplyDelete
  2. robansuini 9:58 AM

    Hey Ana!
    Indeed they have planned to implement the two levels of priorities but it isn't yet available.

    Meanwhile they release this. Managing different kind of queues to handle priorities seems to be a good way to solve the problem. Eventually changing from multiple queues to a single queue with priorities shouldn't represent much effort on my point of view.

    The other advantage that RabbitMQ has is a very nice scalable hability if you are thinking on a cloud deployment.

    The following is a very nice presentation about RabbitMQ describing some of the cool features it has:
    http://www.rabbitmq.com/resources/RabbitMQ_MessagingInTheCloud_VMworld_2010_MS.pdf

    Thanks for writing!

    ReplyDelete
  3. alexis6:05 PM

    A good way to model priorities in AMQP is to use the routing keys to indicate priority. For example messages marked "1" go to queue "1", and messages marked "2" to queue "2" and so on for any N. You can use topic routing to get more sophisticated routing patterns eg "system.event.3"

    alexis

    ReplyDelete
  4. robansuini 9:56 PM

    Hey alexis thank you very much. Indeed that approximates very much to the workaround for managing priorities. There is one thing that keeps bugging me is on how to create a consumer that reads first from those queues with higher priority and then proceed to continue with the ones with lower priority. Or is it better to have several consumers for each queue? which could be more resource expensive.

    ReplyDelete

Post a Comment

Popular posts from this blog

Start-up

After quite a few busy weeks I'm finally back to the blog. The reason for such absence was the change to a new job. And meant a more profound change than just switching jobs and let me explain what I mean with this by listing what has changed: From following to leading From guidelines to freeform From legacy to state-of-the-art From the consultancy world to the entrepreneur world And the last one pretty much sums it up... I recently left the consultancy world to join a "backed-up" start-up company on a moment where many may think it was a very risky moment to do so. Leaving from a established consultancy company to recently created one doesn't seem like something most people would do on times of financial crisis but to me it was worth the risk by believing that crisis times are those times where opportunities open to those few people who take it. It gives you the opportunity to reinvent yourself and to do something new or something better and to fill tha
Read more

JQuery development and Chrome local files access

If you have been developing a web page using JQuery and trying access some local json file. $.getJSON("your path to the local file", function(data){ //some code }); You might have found out that trying to run your code con Chrome it fails and with little detail on why it isn't working. You may have thought that there is a problem with your code or even with the validation of the json data which is an option also. But before going all crazy reviewing all of the code. Check this: Chrome for security reasons doesn't allow local files to access other local files by default. So if you want your code to work there is a simple fix. Start your Chrome browser using the following parameter: --allow-file-access-from-files and thats it your code should be working now and you should be able to work on your json data. If you are using firefox then you should take a look at this link: http://kb.mozillazine.org/Links_to_local_pages_do_not_work
Read more