Disaster Squad: behind the scenes at the Xbox Live Operations Centre


Recommended Posts

What does it take to keep gaming's most popular online service not just alive but thriving? An inhuman amount of work and a staggering level of planning. We make an exclusive pilgrimage to the never-before-photographed home of online gaming's guardian angels: the Xbox Live Operations Centre

If you imagine Xbox Live as a living, breathing organism, the Xbox Live Operations Centre is the nervous system, the first line of defence in maintaining an immense global network that serves millions of users daily. Every technical hiccup, no matter how isolated or insignificant, is instantly identified and attacked with unflinching efficiency by a team of first-responders, the full might of the entire Xbox team just an email away.

 

image_39203_620.jpg

Yet, the command centre itself - abbreviated as the XOC - projects a modesty that belies its grand purpose. Behind an unmarked door in an anonymous building on Microsoft's sprawling Redmond campus, a single windowless room houses the entire operation. The floor is terraced, affording every shift member a clear view of the various screens mounted at the front of the room, just in case the three monitors that populate each workstation prove somehow insufficient. The room is manned 24 hours a day, 365 days a year, and analogue clocks display current times in various parts of the world since, hey, it's always prime time somewhere.

And looming ominously above it all is a massive red digital timer. "Any time there's a major incident going on that's either user-impacting or we think it might be user-impacting, we start that clock," warns Microsoft Gaming Ninja (seriously, that's actually his title) Eric Neustadter. "We use that to make sure we're doing the right things at each point in our process. We have playbooks for every kind of major incident, which tell us what to do at each point in the process. There's actually someone in the room during an incident whose job is incident commander. They don't troubleshoot. Their job is to watch the clock and to follow the process and make sure we don't skip anything."

As if on cue, the clock suddenly starts ticking. Neustadter laughs; apparently this isn't a rare occurrence. "What started that clock could have been one of our monitoring systems alerting - the equivalent of a big red light," he explains. "We've got alerts on essentially anything a customer could do where it's going to tell us if it runs into an error. So when a console tries to do something and fails, it lets us know instantly."

 

image_39200_620.jpg

Of course, given the breadth of the XOC's monitoring duties - which include not only the public face of Xbox Live but behind-the-scenes developer systems as well - there are a variety of other possibilities. "It could have been a report from @XboxSupport or someone else on the Xbox team saying that, 'Hey, something happened. Please go look into it.'" Neustadter continues. "It also could be an external event. It could be one of our partners' data centres just had a complete outage and now users can't play game 'blank.' But our job is to find out, 'Is it ours? Is it external?' Manage the communications, make sure we tell xbox.com, customer support, all those different customer touchpoints what's going on, and make sure that all the right people are engaged."

The process

While the team spends plenty of time working to resolve known issues, just as often, they have to go looking for trouble. "That's what the majority of these people are doing all the time," says Neustadter. "They're watching for any interruptions in the service. Anytime we see anything unusual, we have to chase it down until we understand it, whether it's an actual problem or not, whether it's our problem or someone else's. Until you understand it, you can't say that nothing's wrong. You can guess, but you'll be wrong often enough that we just chase them all down."

More here: http://www.totalxbox.com/73483/features/disaster-squad-behind-the-scenes-at-the-xbox-live-operations-centre/

Link to comment
Share on other sites

This is a great read. I've been on Live for a long long time now (11-12yrs iirc)

And I have rarely, and I mean rarely had a problem with service. Sure issues arise with any network infrastructure, but nothing that ever made me question the service.

Now that I have read what really goes on. It really give a different perspective where $60yr is going. This XOC doesn't run for free (staff, nor machine), so does Live pretty much pay for itself, and it's not "straight profit" for Microsoft? Or at the end of the day, does MS make a few bucks off of it?

"Our backup plans, have backup plans". <------ that's the way it's suppose to be.

Anyways. Keep up the good work Microsoft.

  • Like 1
Link to comment
Share on other sites

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

Link to comment
Share on other sites

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

 

Looks cleaner than most ops centers I've been in. And so what. It's supposed to be functional, not sexy.  I doubt many people outside those who work in that room know where that room is at Redmond anyhow.

Link to comment
Share on other sites

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

 

That's just the false flag location used for PR. The actual place has a door the size of a Titan and as thick as a house, and is run by WOPR :rofl:

Link to comment
Share on other sites

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

You'll love the NSA then :D

 

78d9bf5056feffce7ee854d5c8e13491.jpg?itoNSA_star_trek_room_2.jpg

Link to comment
Share on other sites

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

No cable management, what is it a DC?

 

It looks like a normal operations center with monitors everywhere with high-energy drinks with a lot of people staring at screens. It looks like every operations room I've ever been in and I work at ISPs.

Link to comment
Share on other sites

This topic is now closed to further replies.