« January 2010 | Main | March 2010 »
February 25, 2010
Queuing Tweets with Amazon’s (AWS) SQS
Like many services, we wanted to enable social sharing on Twitter (and other platforms like Facebook) at the launch of PersistentFan.
Initially, our interface with Twitter worked reliably and allowed people to tweet about interesting videos. A basic flow diagram looked like this:
However, once we opened up PersistentFan, the increase in traffic (which wasn’t a tidal wave) we noticed a lot of tweets weren’t showing up on Twitter.
Our first attempt was to simply add a basic retry mechanism, which improved things, but only slightly. Our second attempt was to retry multiple times. The flow diagram turned into something like this:
Unfortunately, several times this led to site performance issues as server threads were busy trying to deliver a tweet instead of handling an inbound HTTP request
I had solved a similar problem (at a much larger scale) at MessageCast by adding message queuing. Implementing a full blown system with ActiveMQ etc seemed like overkill so I took a look at AWS SQS. It was really easy (and quick) to build a basic prototype. After looking at the typica library, I ended up choosing the AWS library and was off and running.
Tweet delivery logic has now changed to:
Overall, things were up and running pretty quickly and we’ve had very few issues. Currently, the AWS console doesn’t allow you to peek into a given queue and manage it (i.e. modify, delete etc) but this is supposed to be added in the near future.
Here’s some basic code for sending and getting to/from a queue:
Sending a message to the queue:
SendMessageRequest request = new SendMessageRequest();
request.setQueueUrl(<QueueURL>);
request.setMessageBody("This is my message text.");
invokeSendMessage(service, request);
Getting a message from the queue:
ReceiveMessageRequest request = new ReceiveMessageRequest();
request.setQueueUrl<QueueURL>;
invokeReceiveMessage(service, request);
Pricing is quite reasonable as well – 10k requests is $0.01
If you’re looking for a queuing solution, I’d recommend taking SQS for a spin.
Posted by davehod at 03:42 PM | Comments (0)
February 16, 2010
Iterating in the Open
Over the years I’ve created, been an Advisor to and invested in online services. There have been various launch strategies from “let’s release as soon as we have these ‘n’ features”, “let’s release when we have feature parity with competitor ‘x’” and “hell, let’s release what we have and iterate as we go”.
There are pros and cons to each of the above three strategies. Based on my experience, the first and third ideas are workable, but waiting for feature parity is equal to never actually shipping anything (I’ve seen this several times).
When Mike and I came up with the idea for what is now PersistentFan (starting as an FB app, “top3Clicks”) we were determined to employ strategy number three, iterating in the open. The vision behind PersistentFan was to create a fan-oriented site where we could create a system that would programmatically acquire content about our various niche (or even micro-niche … is that an actual term??) areas of interest, notify us and enable sharing with our friends. We started off with something easy – video. Obviously, there are other types of content for a given area (news, blogs, photos, audio, etc) but YouTube had great APIs to get the ball rolling.
We aimed to iterate several times a week, pushing new features and bug fixes (here’s a sample). As the weeks flew by, the functionality of the site would increase bit by bit until we had a full featured offering. (Note that we did start off in bare-bones, early alpha invite-only mode). Feedback from friends (initially) and as the site grew, external users would help guide both the features and the priority of the features we shipped. Not crowd sourcing as described in Don Tapscott’s “Wikinomics” but instead open iteration, warts and all.
It’s been several months since we had the first user take the site for a spin and so far, here’s what we’ve found:
The good parts:
- The ability to evolve service based on reality (i.e. user feedback, actual utilization)
- Feedback based on usage instead of looking at a PowerPoint slide
- Users pushing up the priority of a feature (we’ve had the request for a “Forgot My Password” several times now *cough*)
The not so good parts:
- A new visitor to the site may not see value on initial visit and never return (the “is that all there is?” problem)
- Bugs, bugs, bugs. We have a staging environment and plenty of automated tests, but when you’re running fast …
Stuff we’ve learned along the way:
- Iterating in the open is a net positive, but users need information describing updates and bug fixes (blog posts are great for this)
- Don’t forget to mail registered users about updates (*cough*)
- Enable the site with an open feedback mechanism (e.g. uservoice)
- Have a strong grasp on analytics to see actual utilization (e.g. number of signups)
- Have internal metrics as well as external (use Google Analytics) to get a good picture of what users are doing and the conversations about your site.
- Reaffirmed that cloud computing is a fantastic way to scale super cheaply (AWS/EC2)
- Automate as much as you can including unit tests, build and deployment scripts, etc. The time savings and reduction in errors pay off quickly.
Become a PersistentFan if you haven’t already, let us know what works/what doesn’t and look for a steady stream of changes. (Any questions why I’m the Mayor at the local Starbucks?)
Posted by davehod at 05:11 PM | Comments (1)
February 04, 2010
Random Collection of Useful Tools
The effort on PersistentFan has led me to seek out various development/build/performance tools lately.
Some of the ones I found particularly helpful are:
- Minify/YUI Compressor – minimizing the number of HTTP GET requests for CSS and Javascript resources and the size (in bytes) of these resources can increase client-side (browser) perf. Matt Snider has a great post on how both to concatenate and compress resources via Ant.
- JSLint: Great service that analyzes your Javascript for code quality. Even better, RockstarApps has encapsulated JSLint to run as an Eclipse plug-in. Haven’t attempted to get this to run as part of the build (with Ant) yet.
- Page Speed – I tried YSlow (add-on to Firebug) but found Google’s tool to be better overall. Really helpful in determining the best way to increase performance.
Posted by davehod at 03:52 PM | Comments (0)