Amazon Ships AWS SDK for Java

Amazon has shipped an AWS SDK for Java (announcement)

Immediate areas of interest for me are:

  • EC2
  • Elastic Load Balancing
  • S3
  • SQS

Now if they would just ship a version of the Product Advertising API (which lets you search Amazon’s ginormous product DB) for Java – it’s been broken since August 15, 2009 when Amazon started requiring signed headers:

Dear Developer,
We want to remind you that all calls to the Product Advertising API must be authenticated using request signatures by August 15, 2009. Please remember that calls to the Product Advertising API that are not signed will not be processed after August 15, 2009. For help on request signatures, please see the Resources section below.
The Product Advertising API Team.

Their Java library does not support this functionality and has not been updated, leaving Developers to follow long posts like this to try and use the Product API. Seems like Jeff Barr or another Amazon Evangelist would want to fix this problem. I’ve only been asking since November 2009 …

Quora – A better FriendFeed/Aardvark?

Dave was kind enough to send me a Quora invite last week. After a few days of trying out the service, I am already hooked.

Quora has a lot of the elements of two services I love: Friendfeed and Aardvark. As Friendfeed is working towards a slow death, I’ve been on the lookout for a something to possibly replace it (I realize that  Quora does not centralize all my tweets etc etc but it *does* provide a place to engage in dialogue about topics of interest, which was always the true value of Friendfeed.)

I’ve made a n00b mistake or two, confused by some of the UI (like this):

Now I’m hoping they expose an API so I can create a killer Firefox add-on like this (which I just submitted an update for 3.6 and up…waiting for approval)  Maybe I can talk sandosh into helping …

Queuing Tweets with Amazon’s (AWS) SQS

Like many services, we wanted to enable social sharing on Twitter (and other platforms like Facebook) at the launch of PersistentFan.

Initially, our interface with Twitter worked reliably and allowed people to tweet about interesting videos. A basic flow diagram looked like this:

However, once we opened up PersistentFan, the increase in traffic (which wasn’t a tidal wave) we noticed a lot of tweets weren’t showing up on Twitter.

Our first attempt was to simply add a basic retry mechanism, which improved things, but only slightly. Our second attempt was to retry multiple times. The flow diagram turned into something like this:


Unfortunately, several times this led to site performance issues as server threads were busy trying to deliver a tweet instead of handling an inbound HTTP request

I had solved a similar problem (at a much larger scale) at MessageCast by adding message queuing. Implementing a full blown system with ActiveMQ etc seemed like overkill so I took a look at AWS SQS. It was really easy (and quick) to build a basic prototype. After looking at the typica library, I ended up choosing the AWS library and was off and running.

Tweet delivery logic has now changed to:


Overall, things were up and running pretty quickly and we’ve had very few issues. Currently, the AWS console doesn’t allow you to peek into a given queue and manage it (i.e. modify, delete etc) but this is supposed to be added in the near future.

Here’s some basic code for sending and getting to/from a queue:

Sending a message to the queue:

SendMessageRequest request = new SendMessageRequest();
request.setMessageBody("This is my message text.");
invokeSendMessage(service, request);

Getting a message from the queue:

ReceiveMessageRequest request = new ReceiveMessageRequest();
invokeReceiveMessage(service, request);

Pricing is quite reasonable as well – 10k requests is $0.01

If you’re looking for a queuing solution, I’d recommend taking SQS for a spin.

Iterating in the Open

Over the years I’ve created, been an Advisor to and invested in online services. There have been various launch strategies from “let’s release as soon as we have these ‘n’ features”, “let’s release when we have feature parity with competitor ‘x’” and “hell, let’s release what we have and iterate as we go”.

There are pros and cons to each of the above three strategies. Based on my experience, the first and third ideas are workable, but waiting for feature parity is equal to never actually shipping anything (I’ve seen this several times).

When Mike and I came up with the idea for what is now PersistentFan (starting as an FB app, “top3Clicks”) we were determined to employ strategy number three, iterating in the open. The vision behind PersistentFan was to create a fan-oriented site where we could create a system that would programmatically acquire content about our various niche (or even micro-niche … is that an actual term??) areas of interest, notify us and enable sharing with our friends. We started off with something easy – video. Obviously, there are other types of content for a given area (news, blogs, photos, audio, etc) but YouTube had great APIs to get the ball rolling.

We aimed to iterate several times a week, pushing new features and bug fixes (here’s a sample). As the weeks flew by, the functionality of the site would increase bit by bit until we had a full featured offering. (Note that we did start off in bare-bones, early alpha invite-only mode). Feedback from friends (initially) and as the site grew, external users would help guide both the features and the priority of the features we shipped. Not crowd sourcing as described in Don Tapscott’s “Wikinomics” but instead open iteration, warts and all.

It’s been several months since we had the first user take the site for a spin and so far, here’s what we’ve found:

The good parts:

  • The ability to evolve service based on reality (i.e. user feedback, actual utilization)
  • Feedback based on usage instead of looking at a PowerPoint slide
  • Users pushing up the priority of a feature (we’ve had the request for a “Forgot My Password” several times now *cough*)

The not so good parts:

  • A new visitor to the site may not see value on initial visit and never return (the “is that all there is?” problem)
  • Bugs, bugs, bugs. We have a staging environment and plenty of automated tests, but when you’re running fast …

Stuff we’ve learned along the way:

  • Iterating in the open is a net positive, but users need information  describing updates and bug fixes (blog posts are great for this)
  • Don’t forget to mail registered users about updates (*cough*)
  • Enable the site with an open feedback mechanism (e.g. uservoice)
  • Have a strong grasp on analytics to see actual utilization (e.g. number of signups)
  • Have internal metrics as well as external (use Google Analytics) to get a good picture of what users are doing and the conversations about your site.
  • Reaffirmed that cloud computing is a fantastic way to scale super cheaply (AWS/EC2)
  • Automate as much as you can including unit tests, build and deployment scripts, etc. The time savings and reduction in errors pay off quickly.

Become a PersistentFan if you haven’t already, let us know what works/what doesn’t and look for a steady stream of changes. (Any questions why I’m the Mayor at the local Starbucks?)

Random Collection of Useful Tools

The effort on PersistentFan has led me to seek out various development/build/performance tools lately.

Some of the ones I found particularly helpful are:

  • Minify/YUI Compressor – minimizing the number of HTTP GET requests for CSS and Javascript resources and the size (in bytes) of these resources can increase client-side (browser) perf. Matt Snider has a great post on how both to concatenate and compress resources via Ant.
  • JSLint: Great service that analyzes your Javascript for code quality. Even better, RockstarApps has encapsulated JSLint to run as an Eclipse plug-in. Haven’t attempted to get this to run as part of the build (with Ant) yet.
  • Page Speed – I tried YSlow (add-on to Firebug) but found Google’s tool to be better overall. Really helpful in determining the best way to increase performance.

PersistentFan Graduates to Beta

Bacon has a post up about PersistentFan leaving invite-only mode.

We’re not quite ready for primetime, calling it an early Beta instead. Like many things, we have a zillion ideas to try out and will continue to rev the service. Check it out if you have a chance.

Here’s a few sample channels to test drive with:

You can also see what other people are watching

If you are a big fan of Jersey Shore or want to keep up with the latest from the disaster in Haiti, PersistentFan enables you to:

  • Create channels of interest that persistently search for new content
  • Receive email notifications when new content is available (you receive these as they are discovered or on a daily or weekly basis)
  • Share your favorites to Facebook and/or Twitter
  • Keep up with video feeds of Twitter’s trending topics

PersistentFan was built with open source, Java, Linux, Apache, Tomcat, mod_jk, mysql, junit, Struts, AWS (EC2 and S3), Javascript and plenty of

Hooked on FourSquare

Every morning, I wait with baited breath. Pushing the “Check-in here” button on my iPhone, I peer at the screen, waiting for the thing that will instantly make my day.

It happened a few times, but somehow, I lost it. I can’t explain why, but I want it back.

I want to the Mayor of my local Starbucks again.  “AJ F.” stole the title from me last week and I’m left wondering if I will ever get it back. My wife thinks I’m nuts. Bacon officially labeled me a “junkie”.



















It isn’t the super cool geo-location (which apparently Gowalla does better??), nor the info my social graph provides. The bugs/UI might drive some folks nuts (e.g. I go to the same place every morning, but it never shows up in my list. I have to type out “S-t-a-r-b-u-c-k-s” every time, wait for the service to respond with a list of Starbucks and scroll through to find my local place).






I can’t explain it … I guess this means I’m a Foursquare fanboi now. And I’m ok with that.