The Beta of the Mail Beta

Lots of people who’ve worked on the Mail Beta have been posting on their blogs, so I’m going to join in the fun. My work in Kahuna/Mail Beta is generally around two areas: first, the architecture and deployment of the Kahuna servers and second, the beta of the product. In this post, I’m going to talk about a bit about the latter.

To kick off the Kahuna beta, we invited a select group of previous beta users to join our beta to help kick the tires, get some feedback, and see how well (or not well!) our servers ran and the product felt. After people joined the beta, we’ve had all sorts of feedback on how we designed the product to ideas (ranging from great to crazy) for future features. What’s been fantastic about this beta is it’s such an early copy of the code, that our beta testers are helping us co-design the future of the product. From what I can tell, they’re loving to pick apart all of our decisions and give us valuable feedback.

So where does all that feedback go from a tester? There are two primary mechanism. First, there are the newsgroups. I and many members of the product team participate in newsgroups, talking directly with our beta testers, asking them questions and answering theirs. The newsgroup is quite active, with over 1400 posts in the last three weeks. We get to directly interact with our users, which is fulfilling for us since most people like the beta, and I imagine great for the beta testers since they can easily get in touch with people making decisions on the product.

Second, is the site where beta testers enter bugs against the Mail Beta. The bugs that are filed on the Connect site directly go in to our product’s bug database (called Product Studio) where a dedicated group of beta team members scrub the bugs to eliminate duplicates or mark bugs that we’ve already had planned to fix. They also communicate the status of the bugs back to the beta testers. To give you an idea, about 700 bugs have been filed since we started and of that, roughly 250 have made it past our beta team’s scrubbing.

If a bug genuine and genuinely new, we then assign it to a PM on the product team to take a look at, who will do another round of “triaging” (i.e. bug scrubbing) before assigning to a developer to get fixed. It sounds a bit complex and perhaps high in process, but we do it so that we can keep our developers working on the most important features and bugs.

In another post later, I’ll hopefully explain a bit about the other half of what I’ve been doing here: the architecture and deployment of the beta.