Friday, July 9, 2010

Cloud-based Message Queueing and Persistence

I think given the choice, most developers rather deal with synchronous vs. asynchronous processes. It’s just much easier to wait for an immediate response and continue down your path vs. having to queue requests and set up a secondary process to poll for replies, perhaps correlate them to the original requests and then process them. However, since you can’t always control every application you need to work with, you will find that having a place to temporarily park messages for either later processing or pick up is necessary. You some times need UPS to retry delivering your package and some times you rather go pick it up at the distribution center.

As an example, if you ever worked with Salesforce’s Workflow and Outbound messaging feature, you’ve probably had to figure out a way to deal with storing the Outbound Notification from Salesforce while your Great Plains , Oracle EBS,... instance is down for maintenance. Salesforce’s Outbound messaging service will only retry a finite amount of times and then the notification is lost so having a place to temporarily store the notification until your application is back on line can be quite necessary.

For most use cases, you have plenty of options for persisting your messages, from simple database tables to more sophisticated message queueing software. If you’re looking to reduce internal IT costs and headaches, Amazon Simple Queue Service (SQS) can be a good option too. It’s simple to set up, it’s fully managed, and it’s very inexpensive. You have several choices of geographic location for the service, US East/West, EU, and APAC. The API is straightforward enough and since it uses standard Cloud-based protocols (RESTFul and SOAP WS over HTTPS), you can be up and running in no time. 

Amazon SQS uses a distributed architecture with multiple queue nodes and as most Cloud-based services go, it some times takes a little time for all the nodes to sync up and have the expected response to your request. Here are some of the behavior I noticed while playing around.

  • There was almost always a delay (10-30secs) in availability of newly created queues. I would create a queue and then use the list queue service to see if it was there. Same was true when I deleted queues. They showed up in the list of queues for 10-30 seconds after I had a confirmation of deletion.
  • Message retrieval seemed some what random. I would send a number of messages in a row and then try to retrieve them. I would get my messages back but in what seemed random groupings. I never lost a message so that was great news.

Overall, SQS was very reliable and as long as you don’t care about ordered delivery and you don’t that have messages that are > 8KB, it’s a viable Cloud-based choice.

Microsoft’s Azure platform offers a similar service which I haven’t played around with but it looks almost identical to Amazon’s SQS. I guess borrowing is the best form of flattery. The operations seem to be identical (1:1) to SQS and here’s a direct quote from their documentation: 
"A queue can contain an unlimited number of messages, each of which can be up to 8 KB in size. Messages are generally added to the end of the queue and retrieved from the front of the queue, although first in, first out (FIFO) behavior is not guaranteed.“ 
So pretty much the same as Amazon's SQS. It's great when your competition doubles as your PM.

If you’re looking for a Cloud-based message persistence service to complement your Cloud-based integration platform, Amazon SQS and (probably) Microsoft’s Azure both offer worthwhile solutions.