Jump to content



Photo
xbox xbox live xbl operations center xoc

  • Please log in to reply
11 replies to this topic

#1 spenser.d

spenser.d

    Neowinian Senior

  • Joined: 19-December 03

Posted 20 April 2014 - 15:52

What does it take to keep gaming's most popular online service not just alive but thriving? An inhuman amount of work and a staggering level of planning. We make an exclusive pilgrimage to the never-before-photographed home of online gaming's guardian angels: the Xbox Live Operations Centre

If you imagine Xbox Live as a living, breathing organism, the Xbox Live Operations Centre is the nervous system, the first line of defence in maintaining an immense global network that serves millions of users daily. Every technical hiccup, no matter how isolated or insignificant, is instantly identified and attacked with unflinching efficiency by a team of first-responders, the full might of the entire Xbox team just an email away.
 

image_39203_620.jpg

Yet, the command centre itself - abbreviated as the XOC - projects a modesty that belies its grand purpose. Behind an unmarked door in an anonymous building on Microsoft's sprawling Redmond campus, a single windowless room houses the entire operation. The floor is terraced, affording every shift member a clear view of the various screens mounted at the front of the room, just in case the three monitors that populate each workstation prove somehow insufficient. The room is manned 24 hours a day, 365 days a year, and analogue clocks display current times in various parts of the world since, hey, it's always prime time somewhere.

And looming ominously above it all is a massive red digital timer. "Any time there's a major incident going on that's either user-impacting or we think it might be user-impacting, we start that clock," warns Microsoft Gaming Ninja (seriously, that's actually his title) Eric Neustadter. "We use that to make sure we're doing the right things at each point in our process. We have playbooks for every kind of major incident, which tell us what to do at each point in the process. There's actually someone in the room during an incident whose job is incident commander. They don't troubleshoot. Their job is to watch the clock and to follow the process and make sure we don't skip anything."

As if on cue, the clock suddenly starts ticking. Neustadter laughs; apparently this isn't a rare occurrence. "What started that clock could have been one of our monitoring systems alerting - the equivalent of a big red light," he explains. "We've got alerts on essentially anything a customer could do where it's going to tell us if it runs into an error. So when a console tries to do something and fails, it lets us know instantly."
 

image_39200_620.jpg

Of course, given the breadth of the XOC's monitoring duties - which include not only the public face of Xbox Live but behind-the-scenes developer systems as well - there are a variety of other possibilities. "It could have been a report from @XboxSupport or someone else on the Xbox team saying that, 'Hey, something happened. Please go look into it.'" Neustadter continues. "It also could be an external event. It could be one of our partners' data centres just had a complete outage and now users can't play game 'blank.' But our job is to find out, 'Is it ours? Is it external?' Manage the communications, make sure we tell xbox.com, customer support, all those different customer touchpoints what's going on, and make sure that all the right people are engaged."

The process

While the team spends plenty of time working to resolve known issues, just as often, they have to go looking for trouble. "That's what the majority of these people are doing all the time," says Neustadter. "They're watching for any interruptions in the service. Anytime we see anything unusual, we have to chase it down until we understand it, whether it's an actual problem or not, whether it's our problem or someone else's. Until you understand it, you can't say that nothing's wrong. You can guess, but you'll be wrong often enough that we just chase them all down."


More here: http://www.totalxbox...rations-centre/




#2 Showan

Showan

    Neowinian Senior

  • Joined: 28-November 12
  • Location: Amurrika
  • OS: W8
  • Phone: Lumia Icon

Posted 20 April 2014 - 18:02

This is a great read. I've been on Live for a long long time now (11-12yrs iirc)
And I have rarely, and I mean rarely had a problem with service. Sure issues arise with any network infrastructure, but nothing that ever made me question the service.

Now that I have read what really goes on. It really give a different perspective where $60yr is going. This XOC doesn't run for free (staff, nor machine), so does Live pretty much pay for itself, and it's not "straight profit" for Microsoft? Or at the end of the day, does MS make a few bucks off of it?

"Our backup plans, have backup plans". <------ that's the way it's suppose to be.

Anyways. Keep up the good work Microsoft.

#3 OP spenser.d

spenser.d

    Neowinian Senior

  • Joined: 19-December 03

Posted 20 April 2014 - 22:47

Agreed, it's always neat to see what goes on behind the scenes. Also puts some more human faces on the service, which I think some people forget about easily.



#4 xendrome

xendrome

    In God We Trust; All Others We Monitor

  • Tech Issues Solved: 13
  • Joined: 05-December 01
  • OS: Windows 8.1 Pro x64

Posted 20 April 2014 - 23:10

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.



#5 OP spenser.d

spenser.d

    Neowinian Senior

  • Joined: 19-December 03

Posted 21 April 2014 - 00:55

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

 

Looks cleaner than most ops centers I've been in. And so what. It's supposed to be functional, not sexy.  I doubt many people outside those who work in that room know where that room is at Redmond anyhow.



#6 neoadorable

neoadorable

    Neowinian Senior

  • Joined: 01-August 05
  • Location: Flyover Country/Pacific Isle

Posted 21 April 2014 - 01:45

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

 

That's just the false flag location used for PR. The actual place has a door the size of a Titan and as thick as a house, and is run by WOPR :rofl:



#7 Ironman273

Ironman273

    Neowinian Fanatic

  • Tech Issues Solved: 5
  • Joined: 26-October 01
  • Location: Florida
  • OS: Windows 8.1 Pro (Work) Windows 10 (Home)
  • Phone: Nokia Lumia 830

Posted 21 April 2014 - 13:32

Would you like to play a game (on Live)?



#8 neoadorable

neoadorable

    Neowinian Senior

  • Joined: 01-August 05
  • Location: Flyover Country/Pacific Isle

Posted 23 April 2014 - 02:41

Would you like to play a game (on Live)?

 

Global Thermonuclear War, only $4.99 on the Store. Easy achievements, too!



#9 Ironman273

Ironman273

    Neowinian Fanatic

  • Tech Issues Solved: 5
  • Joined: 26-October 01
  • Location: Florida
  • OS: Windows 8.1 Pro (Work) Windows 10 (Home)
  • Phone: Nokia Lumia 830

Posted 23 April 2014 - 03:33

Global Thermonuclear War, only $4.99 on the Store. Easy achievements, too!

A strange game...

 

How about a nice http://marketplace.x...15-d8025a4507d1



#10 blerk

blerk

    Banned User

  • Tech Issues Solved: 1
  • Joined: 09-December 10

Posted 23 April 2014 - 04:04

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

You'll love the NSA then :D

 

78d9bf5056feffce7ee854d5c8e13491.jpg?itoNSA_star_trek_room_2.jpg



#11 neoadorable

neoadorable

    Neowinian Senior

  • Joined: 01-August 05
  • Location: Flyover Country/Pacific Isle

Posted 23 April 2014 - 06:34

A strange game...

 

How about a nice http://marketplace.x...15-d8025a4507d1

 

When did this come out? :D Wish it was for X1 though!



#12 JonnyLH

JonnyLH

    I say things.

  • Joined: 15-February 13
  • Location: UK
  • OS: W8, W7, WP8, iOS, Ubuntu
  • Phone: Nokia Lumia 920

Posted 23 April 2014 - 09:18

From the pics, something that makes them millions upon millions of dollars per year, I am super unimpressed with the setup. Basic desks, no cable management, small room, etc. It just doesn't look like any other operations center I have seen or been in. Even small operations.

No cable management, what is it a DC?

 

It looks like a normal operations center with monitors everywhere with high-energy drinks with a lot of people staring at screens. It looks like every operations room I've ever been in and I work at ISPs.