Hodson Blog

Yesterday (6/16/2009) I attended the Amazon “AWS Start-up Tour 2009” in Sunnyvale at the PlugandPlay Tech Center (which reminded me of a larger version of the The Enterprise Network (TEN) Incubator we were in when we started iPrint)

Update: Slides are here

The event centered around cloud computing and how to utilize the various components of AWS. We’ve been running top3Clicks on AWS (EC2/S3) for almost a year and a half so I was pretty familiar with the basic offerings.

Things were kicked-off by Andy Jassy who runs the AWS business (and authored the initial business plan). He gave a good overview of all that AWS has to offer, including their new offerings around scaling, CDN and monitoring.

The best part of the event were the case studies of several companies utilizing AWS for some or all of their production infrastructure:

• Paco Nathan, Principal Scientist, ShareThis
• Ljubomir Buturovic, Ph.D., Sr. Director and Chief Scientist, Pathwork
• Santosh Rau, Engineering Manager, Software Infrastructure, Netflix
• Andrew Gibbons, Director of Operations, Smugmug

ShareThis and Smugmug are running significant portions of their infrastructure on AWS and both are looking at how to increase their utilization of the platform. Pathwork and ShareThis were interesting in that both had batch scenarios where they needed to provision a large number of machines (more than 1000) for a period of time. Once the batch jobs were complete, they would tear down the instances, thereby reducing both their costs and need for hardware.

At least one company was also using AWS to fire up Dev/Test instances on-demand, run scenarios and then tear down the environment. Given that we’ve spent a lot of effort on something quite similar (using Hyper-V for virtualization of test instances) on my current team, I found it extremely interesting how quick/easy this could be accomplished. There wasn’t any detail, but one scenario might go something like:

1. Create daily/nightly RC build
2. Fire up EC2 instance(s) that replicate enough of the production environment to verify the RC
3. Install/deploy nightly RC to cloud
4. Run automated functional suite of tests
5. Generate reports
6. Tear down environment

No need to have a farm of boxes sitting around that are used for a few hours a day, taking up space in a lab or datacenter. Nice!

The final presentation was done by Jinesh Varia, Technical Evangelist for AWS. He walked through how to take an existing classic three-tier infrastructure and port it to AWS. He pushed the Presentation (html, images, etc) to S3, the Application layer to EC2 and the Data tier to SQS. It was a slightly simplistic view of the world (he only had 45 minutes) but left us able to extrapolate more complex cloud-based architectures.

Overall, a great overview of one company’s view of cloud computing. I left wondering again when Amazon (AMZN) will be acquired, not for their retail operation, but as an infrastructure play. Guessing it will be IBM and it will happen in the next year or so, especially given that Sun (SUNW) is going to Oracle (ORCL).

Twitter is great but, as you might have noticed, missing a few features.

Two apps I’ve been using to help round out the Twitter feature set:

• Twimailer – Twitter makes it tough to know when you have new followers. It is even tougher to know if you want to follow them back. Twimailer addresses this by pushing an email to you every time a person starts following you. They provide a fairly rich message, including the person’s profile, recent tweets and number of followers/following.
  A good number of follows I receive on a daily basis are pure spam. Twimailer makes it easy to quickly determine if the follow is legitimate and if I should follow the person back.
• TweetBeep – Great tool to stay apprised of conversations you care about – people compare it to Google Alerts.
  Personally, I use it a bit differently; tired of missing @replies, I set up a search that emails me when people tweet replies to me (@davehod). Super useful, especially when I’m away from my Seesmic client for any length of time.

Last Saturday (5/30/2009), we ran the Newport (Oregon) Marathon.

The race started at 7am at the Yaquina Bay State Park with approximately 800 participants. The temperature was in the high 40s with a good bit of fog.

Starting off, we headed north and ran a 4 mile loop up to historic Nye Beach and back. There were numerous small hills on this part of the course, but nothing serious. We ran back through the State Park and headed South East on Bay Street, following along the Yaquina Bay. The fog started burning off around Mile 6 or so, which was a bit worrisome as the temperature the previous day had been in the mid 80s.

After running the length of the bay, we headed up the Yaquina River. Like the Bay, this part of the run was very scenic/beautiful. We saw an otter, bald eagle and plenty of trees. Much better than running through a business park in Anytown, USA.

At Mile 11, we ran through Oysterville. There were oyster shooters available for anyone interested (apparently someone hit a new course record by having 38 shooters!). I decided to pass up on the opportunity, promising to have one on the return trip at Mile 18.

We hit the turnaround at Mile 15.5 in Toledo. I was feeling great even though the heat was picking up. On the way back to Oysterville, the wind picked up (this was good as it reduced the heat). Unfortunately, it grew to about a 25 mile/hour headwind, which slowed everyone down a bit. At Mile 18, I decided to once again pass on the oysters, instead focusing on getting past the wall at 20.

Miles 20 – 26 were true to form; tough and best run with an iPod. Fortunately, I had loaded mine up with plenty of motivating music to help me along.

There is one last hill at Mile 26, which is not huge, but by this point in a marathon any hill is tough.  Luckily, “Horseshoes and Hand Grenades” by Green Day started playing on my iPod. As soon as Billie Joe snarled the opening line “I’m not f*cking around!” my adrenaline (or what was left of it) started pumping hard. Fired up, the top of the hill with the “Mile 26” sign appeared in my sights. I made it to the top, flew down the other side (steep!) and crossed the finish. Tired, but fired up.

The race FAQ says “almost half our finishers record P.R.’s”. Don’t believe everything you read. I wasn’t even close to a P.R. :-(

Course Map

Overall, here's how the run rates in my book:

Organization – Well-organized/staffed.  Course was clearly marked, mile markers were accurate. Grade: A

Course – Incredibly beautiful, start to finish. Grade: A+

Aid-stations - One complaint: the website says aid stations are “approximately every two miles along the course”. This is actually not the case, especially as there are no aid stations between Miles 20 and 25 (!) and a number of the aid stations were not where the race handouts stated. For example, an aid station was supposed to be at say Mile 8.6, but it would appear instead of Mile 8.1. Grade: B

Swag – Short sleeve, dri-fit shirt. Grade: A

An interesting quote from Ray Ozzie at the J.P. Morgan Tech conference today

“… at some point in time every major enterprise, every company, every ISV is going to have some blend of software that runs on-premises and some that runs in the cloud, and everyone wants tools that they can use to in essence deploy some apps to part of their organization that might be in the cloud …”

(h/t TechCrunchIT)

Nothing revolutionary is his quote, but it does re-emphasize my thoughts about Amazon/EC2 – when are they valued for their cloud computing technology? When are they acquired for their cloud computing technology?

Amazon announced three new features for EC2 today:

• CloudWatch – monitoring your instances
• AutoScaling – automagically scale/shrink your instances
• Elastic Load Balancing – balance traffic across instances, including geo-distributed clusters (nice!)

There are a lot of conversations going on about the new features. Looks like Amazon has significantly extended their lead in cloud computing. Who'd a thunk it?

Several things jump to mind for me:

* BaconMarathon – we've been running our top3Clicks Facebook app in the EC2 cloud for more than a year now. Overall, the experience has been quite good, but there were a few things missing, most notably a scaling technology solution. I looked at RightScale, but their price point was not for us. Additionally, basic monitoring (disk space, CPU, RAM, app-level like HTTP GET) would have saved us from a few outages and the heartburn that comes with it.

* AMZN p/e - when does Amazon's p/e ratio begin to reflect the technical leadership/lead that they have established in cloud computing? Or has this already occurred since $AMZN is trading at a current p/e of 48.64 while GOOG is 29.02? I saw just today that AT&T was launching a cloud initiative with EMC; great that they are coming to the party, but they are pretty late. Amazon is grouped in Retail - when does this change/does Amazon need to spin this out into a separate company to realize the true forward looking value of the technology? Or, when does an IBM acquire Amazon for the technology and sell-off the retail business? Even more far-fetched, when does Amazon start acquiring technology companies in an Oracle-type rollup strategy?

Great comment from Matthew Snape on my Friendfeed post:

Dave AWS is so brilliant because it is a retail operation. They sell CPU time in exactly the same way they sell books or dvd's, all you need is a credit card. Compare that with the way IBM or Oracle operate. Perhaps it will be Amazon buying Oracle?

* Startups - We've all seen this movie before, but when startups build functionality on top of another company's APIs, things usually go something like this:

• Startup sees APIs, builds cool/missing functionality to fill in the gaps. Ignores comments that they are "just a feature"
• Startup secretly hopes API publisher sees value and acquires startup
• API publisher sees that startup has validated model and builds out same functionality
• API publisher launches new features; startup ends up being "just a feature" and closes down

Perhaps folks like RightScale will figure out how to add more value/build out their cloud support to other offerings like IBM, Google, Microsoft, etc.

Huzzah!

I’m a little behind, but apparently the USATF has decided to give marathon race directors the leeway to allow headphones during races. No more sneaking in your iPods, risking a DNF after all your hard training. Or only running marathons that choose to go against the USATF.

Excellent news, as I’m definitely bringing my iPod Shuffle when we run the Newport Oregon marathon in 12 days (<gulp>). Gotta load it up with the new Green Day…

Last Thursday night, Bacon heard from a top3Clicks user that our app was broken and provided some specifics repro cases. We checked them out, and sure enough the user was correct. Our app on Facebook was actually unusable, a true high priority situation.

Thanks to our easy roll-forward/roll-back deployment mechanisms, I was able to quickly run through the last 4 builds, diff all the JS (in the actual cloud and in SVN) and determine that *we* didn’t break anything.

So I started searching the FB dev forums.

A few minutes later, I found it: an FB deployment on Thursday contained a critical bug. The particular bug was disastrous for our app and many others as it broke any use of Javascript EventListeners. Other developers posted repro cases, eventually getting down to this simple case:

Just wanted to add that simply calling purgeEventListeners on an element is enough to notice that it doesn't work

A case this simple provides ample evidence that Facebook is doing a poor job of unit testing their changes. There are a number of ways to unit test Javascript in general, including JsUnit.

In this day and age, there is no excuse for not requiring developers to create/maintain an adequate level of unit tests. In fact, the absence of unit tests is a reason *to not ship*. The challenge of quick iterative cycles should not include a culture where having your end-users function as your QA is standard operating procedure. There is a wealth of information available on how to be nimble while producing high-quality output.

If anyone has links to FB presentations about Developer quality processes/metrics, please send them my way.

Update:
The fix is now broken again. Guess they rolled back the change. Sheesh

So many of us thought Twitter had truly jumped the shark a few weeks ago.

Was it the Ashton Kutcher race with CNN to get to a million followers? Or the very high profile Oprah sign-up/usage on national TV. Maybe it was all the follow-up stories about how 60% of new Twitter users quit after a month.

Perhaps.

Yesterday though, it became official. The other shark-jumping signs were just head fakes compared to this one: Steve Gillmor (of Attention Xml fame – yeah, remember that one?) posted on TechCrunch that RSS is dead (“Rest in Peace, RSS”). Who exactly killed RSS? Why Twitter of course.

Yes, those 140 characters of wonder have replaced any and all value that RSS brings. Steve argues that the real-time nature of Twitter is so superior that Fred Wilson’s latest Boxee post just smokes in Twitter but sucks in RSS.

Steve, get real.

Twitter is great. RSS ain’t dead. Long live RSS (and Atom :-))

Fred Wilson has an interesting post up on the soon-to-arrive market for selling non-public company stock. He’s written about this previously; overall it is a very interesting idea. Fred has some interesting data on the secondary market for Facebook stock (which is still private)

VCs need some type of exit to invest. Entrepreneurs need VCs for funding. No exits ends up meaning no ecosystem. Great quote:

Entrepreneurs won’t start companies and investors won’t invest in them if there is no path to liquidity on the company stock. A secondary market for private company stock can fill the gap that the lack of an I.P.O. market has created.

This does bring to the fore a number of issues around retention and keeping the early-in employees motivated (in my experience, the early-in employees are highly motivated people that are “smart and gets things done”). It’ll be interesting to see how it all plays out.

I was recently asked to produce a two-slide deck, discussing some easy ways to reduce team friction and encourage management-level collaboration.

My findings won’t knock your socks off.

Seriously though, I’m continually amazed how many teams I’ve encountered that don’t do what I’d consider "the basics" of managing team deliverables and expectations. They stumble along, can't understand why each project is so stressful, are shocked at team attrition levels and seem to think "last minute heroics" are a good thing (instead of realizing that many times this is just an indicator of poor planning).

My slides were as follows:

Part One: Project Management

• What/Why – state the goals/objectives of the project

• Who – list out the specific tasks and resources for each task

• How – produce peer-reviewed specifications

• When – create/maintain/publish delivery schedules. Hold individuals accountable.

Part Two - Collaboration

• Hold a weekly meeting consisting of the leadership team (varies by project/team). Each person should discuss current projects, status, set expectations

• Produce a weekly project status to the entire team. Stress accountability

• Ensure product, design and test specifications are peer-reviewed.

• Broadcast any changes to the plan

If you're team isn't doing the above, give it a shot. You'll be amazed at the improvement in team dynamics.