Over the weekend T-Mobile and Microsoft Danger confirmed a huge outage on its Sidekick devices.In a statement issued to customers T-Mobile confirmed "we must now inform you that personal information stored on your device - such as contacts, calendar entries, to-do lists or photos - that is no longer on your Sidekick almost certainly has been lost as a result of a server failure at Microsoft/Danger."
According to some reports, the failure was due to a SAN (Storage Area Network) gone wrong at Microsoft's end. It is claimed that Microsoft does not have a working backup of some of the data that has gone missing from customers devices. The SAN upgrade is rumoured to have been outsourced to Hitachi to complete.
This huge failure has affected hundreds and it's clear from some posts at the T-Mobile forums that customers are both confused and angry at what has happened. One user states "I just want to give hell to whom is responsible for this."
The phone itself does keep a copy of the data locally, but its only stored in RAM, and when you turn the phone off, all the data is synchronized back to their servers and lost locally.
You can bet Hitachi told them they didn't need to back up the SAN, and that they could upgrade the SAN online while not losing any data, and that they had done it "hundreds of times" without any problems.
They would have been under pressure from Microsoft/Danger to do the upgrade with "no downtime" to the service.
Same old story - I've seen Hitachi do it before with exactly the same results, and then blame "out of date firmware", "we've never seen that before", "it was a once in a million year event".
They would have been under pressure from Microsoft/Danger to do the upgrade with "no downtime" to the service.
Same old story - I've seen Hitachi do it before with exactly the same results, and then blame "out of date firmware", "we've never seen that before", "it was a once in a million year event".
Ouch... Sounds like they need to change their practices...
=========================
Eventually someone was going show up the vulnerability of blind faith in "the cloud", I'm just glad it wasn't something that important. Yes, this is a disaster, but it could be a lot worse. I mean lets face it, if Gmail (or another online storage mechanism of equal size) fell over this spectacularly it would be cataclismic.
That said, Danger had better have some cash at the ready, because there's going to be some hefty settlements, its extremely lucky that they've got the infinitely deep pockets of Microsoft to back them up.
You're telling me with a straight face that a company who spends $500 million doesn't care about how the business operates day-to-day?
Microsoft (I'm assuming, given that they're a smart business) own the company, and it allows them to utilise their resources, and in exchange MS make money from it, but the running of Danger as a company remains fairly autonomous.
sure you did.
Become a parent, or an adult, and then you'll realize what the difference between blame and responsibility is. When a kid kicks a football through a window, its not the parents fault, but its their responsibility to deal with it.
Get what I'm saying now?
You know what really makes me laugh (or cry, I haven't decided which yet)? The fact that I mentioned Microsoft completely blinded people to the point of my ACTUAL comment.
This is an EPIC FAIL for not keeping proper backups. The cloud is as good as the brains that run it.
Stupid brains = Fail Cloud
Stupid brains = Fail Cloud
HAHAHAHA
OK a bit overboard I know.......
You know....being a moron and all.
Wait a sec, this sounds like a movie....humans out of the loop....computers making decisions....end of the world....Terminator!!
Living dangerously is the spice of life
Sounds like insufficient computing, storage, nd backup resources were dedicated to this sidekick "cloud" service. Couple this with poor change control at Microsoft (we are all familiar with the quality of their products) and you can see how a poorly run operation can run into trouble.
Having vendors in your environment is a fact of life in the modern data center, weather they are HP, IBM, EMC, Hitachi, or NetApp is not the issue.
The issue is lack of sufficient protection of customer data and disaster recovery planning in case of an event. Disaster events can include power failures, flooding, fire, human error, tornado's, human error in the data center, etc..
I would not trust my data to a compny called Danger...sounds like they have not thought out their "cloud" business model. Who uses TMobile anyway...I like Verizon myself.
They should have tested it with a simulator by creating a virtual lab with all cloud hardware and servers.
If they had done this, they would have found the issue very quickly before going public with it.
Commenting has either been disabled on this article or you are not logged in. Click here to login or register, its free!
Note: Anonymous commenting is disabled in order to keep the quality of responses to a high standard.