Hotmail Windows Migration Case Study

There was a case study written a few years back about Hotmail’s migration from FreeBSD to Windows 2000 that provides a facinating insight in to Hotmail as a service as well as how Hotmail works today. The numbers are a bit dated in the document, but still the right scale (over half a million chanages a day between user adds and deletes, thousands of front end servers) as is the description of our front end architecture (web servers, DAV servers, login servers, etc). A lot of the document is also really interesting as it talks about some of the performance and productivity gains that were seen during the move to Windows 2000 (like the private heap that prevents memory leeks, the Perl legacy of the code base (which is why the front end code base has it’s own STR class), the use of Unix tools, like grep/syslog/etc, on a front door and more). Neat stuff that’s a great historical perspective of the Hotmail front end.

Email Is Nearly Unbearable

No, I’m not talking about Hotmail, just my work email. I’ve just caught up on email from the weekend (yes, it’s Monday night). I have a Search view in Outlook that shows me what email has crossed my path today (cleverly enough called “Today’s Mail”) that I use to keep track of threads. Checking it right now it shows 413 mails, meaning 413 mails have crossed my eyes (it takes out mail from listservs I’m on as well as removes my deleted mail). Of that, I sent 114 mails. My inbox is finally down to 44 unread mails out of a total of 338. I’ve had to set my auto-archive to only keep three weeks worth of mail on the server and yet I still run out of space nearly every two or three weeks. Gaa, this is getting crazy!

Beat The Halo 2 Campaign

Jamus and I played Halo for another eight or so hours this weekend and beat the campaign in Halo 2. While it was quite fun, I was a bit dissapointed. The story was good and I enjoyed the plot, but the story telling left something to be desired. Lots of the scenes were not stiched together well and the amount of switching between the Covenant and the humans left parts of the plot confusing. Worst of all was the cliffhanger ending. While I’ll be happy to buy Halo 3, I wanted some more closure to the plot line in this one. Jamus pointed me at some propoganda online on how they scaled back Halo 2, as it was supposed to feature 16 levels, 4 on Halo, 4 on the Covenant home world and 8 on Earth. Halo 2 delivered a whopping one on Earth, so it looks like they scaled it back in interest of either shipping (understandable) or making a Halo 3. Great gameplay and I enjoyed every minute of playing, just wanted more.

We Are PAID USER A

PAID USER A – contact XXXXX for questions

My name is Maximus Decimus Meridius, Commander of the Armies of the North, General of the Felix Legions, loyal servant to the true emperor, Marcus Aurelius. Father to a murdered son, husband to a murdered wife. And I will have my vengeance, in this lBurger?

“Now thats a fire!”
“Ice cream!”

For over a thousand years Roman conquerors returning from the wars enjoyed the honor of triumph, a tumultuous parade. The conquerors rode in a trimphal chariot, the dazed prisoners walking in chains before him.Pizza?
Sometimes his children robed in white stoodwith him in the chariot or rode the trace horses. A slave stood behind the conqueror holding a golden crown and whispering in his ear a warning: that all glory is fleeting.Burrito?

Some users may have noticed the message above when they logged in to Hotmail yesterday afternoon, wondering what’s up? Perhaps an early April Fools joke? Maybe Hotmail was hacked? Nothing that exciting, I’m happy to report. Yesterday we shipped a new version of our site, with upgrades such as Photo Mail, stronger Messenger integration, a new hotmail.co.uk domain among other things. But not on the feature list was a speech from Gladiator. So, what’s the deal?

When we released the new version of the code for the Hotmail release, by accident we left enabled a test page with test content, producing the hilarous message you see above that included the email address of a tester to contact during our internal testing (who has since been bombarded with email from all over, including a resume). We noticed the “issue” about 10 minutes after the code went live to the site. Thankfully, during a release, everybody from every walk of life gets together in a room with laptops in hand and monitoring on projectors, much like the well documented Windows ship room, called a release rally. Here is a picture from it:

 

During the rally people are on point to make decisions and be reactive to any issues that might come up (shipping software that hits hundreds of millions of people can cause the craziest things to happen), so as you imagine when the test content went out to the site, people started scrabling and did an great job of finding the root cause and making a fix. The end game and release day are the most fun and invigorating parts of shipping software — decisions happen real time, tensions run high and you’re on the line for delivering a service for millions of users. We also got coverage from Neowin and based on the comments on the site, people got a good laugh out of it, like a bunch of us here did as well.

lBurger?

Work Can Be Hilarous

I had the funniest thread at work today. We’re trying to debug a problem with some code we use, internal to Microsoft. We’re not sure if it’s in one team’s code or another’s, but we’re certain it’s not ours and we have log files to show it. So, we passed on the logs to a thread that had both the teams on it saying, “check it out, what’s up?” After a couple of emails on the thread going, “Hum, this doesn’t look like it’s our fault” and “I don’t get it, the other code is going to send us something, but then decides to give up” and after a couple titrations it gets settled that that it’s one of team’s issue. While that may not sound very funny, it’s hilarious banter to see dev’s both getting completly flumoxed and tossing the hot potato from one side to the other. After writing this up, it’s probably not that funny all, but I’m still grinning.

Back in the States

I’m back in California after being out of the country for a week and a half. Can’t really say I’m that excited to be back. It’s raining outside, the weather is miserable, Bush is still president and I have a pile of work to dig myself out of. Halo 2 is out, so I’ll be happy to waste my weekend away playing make believe as the Master Chief. Also, today is Diwali, and I talked to my parents and grandparents in India, which is always a good thing. Thanksgiving is two weeks away and the holidays are right around the corner; December holds vacationing in Costa Rica plus and extended trip home.

Dublin was fantastic. I went and visisted Trinity College, the Guinesses Storehouse, St. Patrick’s Cathederal among other things. I had a fablous dinner with the Ireland team at One Pico as well as plenty of Guiness and whiskey. On the plane ride back to the states I met two very cool Irish girls and we talked through the entire flight on the way back, ignoring the movie in exchange for conversation. Crazy thing about coming back from Dublin was that I went through US imigration in the Dublin airport. There was a US Border Control station in the Dublin airport and my passport is stamped “Admited – DBL”. That’s the only time I’ve ever seen that happen. Pictures of Dublin.

A Long(er) Update After A Quick Update

I’m in Dublin hanging out in my hotel room working on mail. Here’s the Reader’s Digest day by day:

  • Wednesday — in the morning went to Threadneedle Street and the City, saw the museum for the Bank of England. Work, met my cousin for dinner in Egham, then a bar with Brian. Pictures.
  • Thursday — saw my dad in the morning (6AM!) at Gatwick Aiport while he was en-route to India, went over to Canary Wharf and then Greenwich for the remaning morning. Lunch with the team, work, had dinner with Minna, then a bar with Brian. Pictures.
  • Friday — shopped at Carnaby Street, arrived at Luton a few hours early since I read my eTicket wrong, got to Paris, went to a party at my sister’s friend’s place, and at 1:30 decided to go clubbing to Caberet, home at 5:30. Pictures.
  • Saturday — around Paris, the Iles, hanging out with sister and friends. Pictures.
  • Sunday — more of Paris, hanging out with sister’s friends, flight to Dublin. Pictures.
  • Monday — meetings in Dublin, dinner with Ireland folks, back at hotel (Westin College Green)

Summary: Paris is lovely but the metro and RER is dirty as can be (on my RER ride from CDG to Chatlet, some guy was throwing up on the train and his friend was spitting on the floor), London is awesome but far too expensive and lacks Paris unparalleled beauty (just look at the pictures!). I’d live in either in a heartbeat. Dublin requires exploration tomorrow.

A Quick Update Before A Long(er) Update

I’m in Paris CDG waiting for my flight to Dublin, frantically trying to catch up on email before my wireless connection times out and my battery dies. Last week in London was great, good times with friends and excellent progress made with work. Paris, as usual, was wonderful. I arrive in Dublin tonight and will blog more details and post pictures once I’m online at the hotel.