T-Mobile and Microsoft Danger Sidekick outage, lost data

Over the weekend T-Mobile and Microsoft Danger confirmed a huge outage on its Sidekick devices.

In a statement issued to customers T-Mobile confirmed "we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger."

According to some reports, the failure was due to a SAN (Storage Area Network) gone wrong at Microsoft's end. It is claimed that Microsoft does not have a working backup of some of the data that has gone missing from customers devices. The SAN upgrade is rumoured to have been outsourced to Hitachi to complete.

This huge failure has affected hundreds and it's clear from some posts at the T-Mobile forums that customers are both confused and angry at what has happened. One user states "I just want to give hell to whom is responsible for this."

Report a problem with article
Previous Story

Twitter adds video support. What!? No it doesn't!

Next Story

Sky Songs to take on iTunes

29 Comments

Commenting is disabled on this article.

Looks like they have not been testing their software and hardware before deployment.
They should have tested it with a simulator by creating a virtual lab with all cloud hardware and servers.
If they had done this, they would have found the issue very quickly before going public with it.

Yes. We need to figure out a way to get humans out of the picture once and for all. Damn humans!

Wait a sec, this sounds like a movie....humans out of the loop....computers making decisions....end of the world....Terminator!!

Ah well. I have no sympathy for the morons that have Sidekicks. They're terrible phones, and storing basic stuff like contacts on a cloud server is beyond pointless. All the other phones out there store their data on the phone itself, couldn't the guys at Danger see that local storage is the best way?

Havent you heard - the cloud is where everything is going. Need to save documents use the cloud, your medical records will be in the cloud, ordering pizza will be in the cloud, the cop that normally runs the speed trap will be in the cloud....the cloud is going to save humanity.....then the cloud will blue screen and humanity will be lost in the cloud.
OK a bit overboard I know.......

Yeah that's why even though I occasionally use online services like Reader and GMail, I always keep a local copy of everything and back up frequently.

Well remember the morons that use the Sidekicks are probably unaware that their phone kept its data in the cloud.

You know....being a moron and all.

mcbazza said,
This is an EPIC FAIL for "in-the-cloud" data/storage.

This is an EPIC FAIL for not keeping proper backups. The cloud is as good as the brains that run it.

Stupid brains = Fail Cloud

r1ddl3r said,
This is an EPIC FAIL for not keeping proper backups. The cloud is as good as the brains that run it.

Stupid brains = Fail Cloud

HAHAHAHA

No, but T-Mobile are using server's that are owned by Danger to store their customers data, I assume because they thought it would be backed up reliably.

Firstly, I want to point out that Microsoft have little to nothing to do with this, this is entirely Danger's fault for not keeping working backups (Cloud Computing 101 sort of stuff), MS may own the company, but they don't have any real input on the company's day-to-day operation.

=========================

Eventually someone was going show up the vulnerability of blind faith in "the cloud", I'm just glad it wasn't something that important. Yes, this is a disaster, but it could be a lot worse. I mean lets face it, if Gmail (or another online storage mechanism of equal size) fell over this spectacularly it would be cataclismic.

That said, Danger had better have some cash at the ready, because there's going to be some hefty settlements, its extremely lucky that they've got the infinitely deep pockets of Microsoft to back them up.

Do you know how much Microsoft paid for Danger? $500 million.

You're telling me with a straight face that a company who spends $500 million doesn't care about how the business operates day-to-day?

I didn't say they didn't care, but they didn't buy them because they were struggling now did they? Why would they replace the people that run the company with Microsoft Monkeys when they were doing well when they bought them.

Microsoft (I'm assuming, given that they're a smart business) own the company, and it allows them to utilise their resources, and in exchange MS make money from it, but the running of Danger as a company remains fairly autonomous.

When a company fails, is is the responsibility of it's owners. Just because the owner does not have management control of the company does not mean that they are not responsible (don't get this confused with liability, Danger may be a Limited Liability Company).

Majesticmerc said,
I didn't say it wasn't their responsibility.

sure you did.

Majesticmerc said,
Firstly, I want to point out that Microsoft have little to nothing to do with this...

Don't take what I said out of context. For you I'll reiterate: Just because they have to take responsibility for the incident doesn't mean it was their fault.

Become a parent, or an adult, and then you'll realize what the difference between blame and responsibility is. When a kid kicks a football through a window, its not the parents fault, but its their responsibility to deal with it.

Get what I'm saying now?

You know what really makes me laugh (or cry, I haven't decided which yet)? The fact that I mentioned Microsoft completely blinded people to the point of my ACTUAL comment.

Ouch... Time for T-Mobile to make some money on those 'Service Level Agreements', and then loose it all when their customers abandon ship ...

Why would such information be stored in the server in the first place? Shouldn't it be storing the data on the phone?

I think the point was to have online synchronisation so you could effectively use your phone to send SMS and the like online.

The phone itself does keep a copy of the data locally, but its only stored in RAM, and when you turn the phone off, all the data is synchronized back to their servers and lost locally.

WhoTheF said,
Why would such information be stored in the server in the first place? Shouldn't it be storing the data on the phone?

You can bet Hitachi told them they didn't need to back up the SAN, and that they could upgrade the SAN online while not losing any data, and that they had done it "hundreds of times" without any problems.

They would have been under pressure from Microsoft/Danger to do the upgrade with "no downtime" to the service.

Same old story - I've seen Hitachi do it before with exactly the same results, and then blame "out of date firmware", "we've never seen that before", "it was a once in a million year event".

dvb2000 said,
You can bet Hitachi told them they didn't need to back up the SAN, and that they could upgrade the SAN online while not losing any data, and that they had done it "hundreds of times" without any problems.

They would have been under pressure from Microsoft/Danger to do the upgrade with "no downtime" to the service.

Same old story - I've seen Hitachi do it before with exactly the same results, and then blame "out of date firmware", "we've never seen that before", "it was a once in a million year event".

Ouch... Sounds like they need to change their practices...