• Sign in to Neowin Faster!

    Create an account on Neowin to contribute and support the site.

Sign in to follow this  

British Airways flight disruption was caused by someone unplugging the power

Recommended Posts

jnelsoninjax    12,585

If you work in technology there’s often a joke about someone tripping over the power cord whenever a server goes down. It appears that joke became reality for British Airways last weekend. British Airways flights were disrupted worldwide due to a power supply issue in the company’s main datacenter, with 75,000 passengers affected by canceled flights. The Times reports that the datacenter shutdown was triggered by a contractor accidentally switching off the power supply. British Airways was forced to cancel all flights from Heathrow and Gatwick in London over a popular public holiday weekend, as all of the company’s IT systems were affected. An investigation will now take place, but will likely focus on human error rather than equipment failure. British Airways will also have to answer why the company wasn’t better prepared for power supply issues, especially after being accused of cost-cutting and outsourcing IT jobs. The Guardian reports that British Airways now faces a compensation bill of more than £100 million ($128 million) for the chaos caused by the power supply problems.

Source

Share this post


Link to post
Share on other sites
+warwagon    13,403

Really? Billed for Chaos? It's not their fault people are animals.

  • Like 1

Share this post


Link to post
Share on other sites
+Human.Online    8,709

And why are safeguards not in place to stop this?  Having been in many datacentres, I can say I've never seen an accessible power socket or switch that didn't at least have a label.

  • Like 1

Share this post


Link to post
Share on other sites
PsYcHoKiLLa    2,481

The culprit...

 

cleaner+hoovering.jpg?format=300w

  • Like 5

Share this post


Link to post
Share on other sites
boo_star    413
35 minutes ago, warwagon said:

Really? Billed for Chaos? It's not their fault people are animals.

They're being billed for delayed flights, not "chaos."

 

But that's a bit too mundane of a reason for most news sites nowadays.

Share this post


Link to post
Share on other sites
dipsylalapo    1,840
On 2017-6-2 at 4:35 PM, Nefarious Trigger said:

And why are safeguards not in place to stop this?  Having been in many datacentres, I can say I've never seen an accessible power socket or switch that didn't at least have a label.

To me it's not even about having safeguards, let's be honest, accidents can happen. What I've been asking is why wasn't there a fail safe. You'd think a critical system like this would have a secondary site and be highly available. 

 

Like you, I don't think I've ever been to a DC where as you said it's difficult to accidentally knock the power out but even if that happens they have backup generators.

  • Like 1

Share this post


Link to post
Share on other sites
Mando    5,117

the report is too simplistic, its not just an unplugging of a power socket, the UPS failed and combined with the generator and secondary backup systems caused a surge when both deployed power to the DC at the same time, so instead of 240V AC it pushed 480V AC (generator plus UPS system) causing a system meltdown across the entire datahall. This does not explain why their other Data centers across town also failed at the same time, they are not connected to the same power. The UPS systems in a datacenter are only there to supply power until the backup generators kick in (20 seconds to 2 minutes tops), then should drop to passive mode (voltage regulation), they didnt. (probably due to lack of maintenance)

 

so no, im afraid "unplugging a pocket socket" didnt cause it. Total lack and care of testing their DR procedures and backup power systems caused it, end of, lack of due diligence combined with not holding spare parts of the fault generator in the uk or even the EU, the parts had to come from the far east.....to save money....heh they now reap what they sow.

 

The worker thats being fingered for it, is an outsourced Datacenter engineer, a scapegoat to cover the real reason that they didnt take DR or biannual testing serious, it also does not explain why their other datacenter was hit with the same issues at the same time. In any Data center (from Tier 1 to 4) the power distribution systems are under lock and key, with only the head electrical engineer and Datacenter Manager having the keys.

 

BAs CEO is lying through his teeth to the "root cause" to save his own sorry ass.

 

https://www.theregister.co.uk/2017/06/02/british_airways_data_centre_configuration/

Edited by Mando
  • Like 3

Share this post


Link to post
Share on other sites
FloatingFatMan    19,251

Yeah, there's absolutely NO way at all this was down to someone disconnecting the power.  Too many fail safes involved for that nonsense.  If I had to guess, I'd say the root cause of all of this was cutbacks.. Something the CEO has done at other airline companies he's run, and that ALL have had problems!

 

  • Like 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.