Like many services, we wanted to enable social sharing on Twitter (and other platforms like Facebook) at the launch of PersistentFan.
Initially, our interface with Twitter worked reliably and allowed people to tweet about interesting videos. A basic flow diagram looked like this:
However, once we opened up PersistentFan, the increase in traffic (which wasn’t a tidal wave) we noticed a lot of tweets weren’t showing up on Twitter.
Our first attempt was to simply add a basic retry mechanism, which improved things, but only slightly. Our second attempt was to retry multiple times. The flow diagram turned into something like this:
Unfortunately, several times this led to site performance issues as server threads were busy trying to deliver a tweet instead of handling an inbound HTTP request
I had solved a similar problem (at a much larger scale) at MessageCast by adding message queuing. Implementing a full blown system with ActiveMQ etc seemed like overkill so I took a look at AWS SQS. It was really easy (and quick) to build a basic prototype. After looking at the typica library, I ended up choosing the AWS library and was off and running.
Tweet delivery logic has now changed to:
Overall, things were up and running pretty quickly and we’ve had very few issues. Currently, the AWS console doesn’t allow you to peek into a given queue and manage it (i.e. modify, delete etc) but this is supposed to be added in the near future.
Here’s some basic code for sending and getting to/from a queue:
Sending a message to the queue:
SendMessageRequest request = new SendMessageRequest();
request.setQueueUrl(<QueueURL>);
request.setMessageBody("This is my message text.");
invokeSendMessage(service, request);
Getting a message from the queue:
ReceiveMessageRequest request = new ReceiveMessageRequest();
request.setQueueUrl<QueueURL>;
invokeReceiveMessage(service, request);
Pricing is quite reasonable as well – 10k requests is $0.01
If you’re looking for a queuing solution, I’d recommend taking SQS for a spin.
