British Airways flight disruption was caused by someone unplugging the power

[jnelsoninjax]

If you work in technology there’s often a joke about someone tripping over the power cord whenever a server goes down. It appears that joke became reality for British Airways last weekend. British Airways flights were disrupted worldwide due to a power supply issue in the company’s main datacenter, with 75,000 passengers affected by canceled flights. The Times reports that the datacenter shutdown was triggered by a contractor accidentally switching off the power supply. British Airways was forced to cancel all flights from Heathrow and Gatwick in London over a popular public holiday weekend, as all of the company’s IT systems were affected. An investigation will now take place, but will likely focus on human error rather than equipment failure. British Airways will also have to answer why the company wasn’t better prepared for power supply issues, especially after being accused of cost-cutting and outsourcing IT jobs. The Guardian reports that British Airways now faces a compensation bill of more than £100 million ($128 million) for the chaos caused by the power supply problems.

Source

[+Warwagon MVC]

Really? Billed for Chaos? It's not their fault people are animals.

[Dick Montage]

And why are safeguards not in place to stop this?  Having been in many datacentres, I can say I've never seen an accessible power socket or switch that didn't at least have a label.

[PsYcHoKiLLa]

The culprit...

 

[boo_star]

35 minutes ago, warwagon said:

Really? Billed for Chaos? It's not their fault people are animals.

They're being billed for delayed flights, not "chaos."

 

But that's a bit too mundane of a reason for most news sites nowadays.

[dipsylalapo Supervisor]

On 2017-6-2 at 4:35 PM, Nefarious Trigger said:

And why are safeguards not in place to stop this?  Having been in many datacentres, I can say I've never seen an accessible power socket or switch that didn't at least have a label.

To me it's not even about having safeguards, let's be honest, accidents can happen. What I've been asking is why wasn't there a fail safe. You'd think a critical system like this would have a secondary site and be highly available. 

 

Like you, I don't think I've ever been to a DC where as you said it's difficult to accidentally knock the power out but even if that happens they have backup generators.

[Mando]

the report is too simplistic, its not just an unplugging of a power socket, the UPS failed and combined with the generator and secondary backup systems caused a surge when both deployed power to the DC at the same time, so instead of 240V AC it pushed 480V AC (generator plus UPS system) causing a system meltdown across the entire datahall. This does not explain why their other Data centers across town also failed at the same time, they are not connected to the same power. The UPS systems in a datacenter are only there to supply power until the backup generators kick in (20 seconds to 2 minutes tops), then should drop to passive mode (voltage regulation), they didnt. (probably due to lack of maintenance)

 

so no, im afraid "unplugging a pocket socket" didnt cause it. Total lack and care of testing their DR procedures and backup power systems caused it, end of, lack of due diligence combined with not holding spare parts of the fault generator in the uk or even the EU, the parts had to come from the far east.....to save money....heh they now reap what they sow.

 

The worker thats being fingered for it, is an outsourced Datacenter engineer, a scapegoat to cover the real reason that they didnt take DR or biannual testing serious, it also does not explain why their other datacenter was hit with the same issues at the same time. In any Data center (from Tier 1 to 4) the power distribution systems are under lock and key, with only the head electrical engineer and Datacenter Manager having the keys.

 

BAs CEO is lying through his teeth to the "root cause" to save his own sorry ass.

 

https://www.theregister.co.uk/2017/06/02/british_airways_data_centre_configuration/

Edited June 4, 2017 by Mando

[FloatingFatMan]

Yeah, there's absolutely NO way at all this was down to someone disconnecting the power.  Too many fail safes involved for that nonsense.  If I had to guess, I'd say the root cause of all of this was cutbacks.. Something the CEO has done at other airline companies he's run, and that ALL have had problems!