Details emerge on earlier Facebook outage

Earlier today many of you probably noticed, and perhaps even posted about it here on the forums, that the social networking giant, Facebook, was down. The outage, which lasted approximately 2.5 hours, according to Robert Johnson of Facebook, hit globally. From feedback on the forums generally stated one of three scenarios: Facebook was completely down, Facebook was extremely slow, or Facebook was relatively normal. The first two options seemed to be the more widespread of the scenarios.

Johnson writes that the cause was an error in handling, ironically, an error condition. Facebook has an automated system to verify configuration values, which checks for invalid entries in the cache, and swaps them with updated values from the persistent store on the site. Unfortunately, when Facebook modified the persistent copy of one configuration value, it was perceived by the automated system as "invalid." When a user tried to access Facebook, that sent a query to the databases attempting to fix the "invalid" value. This wouldn't have been such an issue, except after the problem had been fixed, the numerous amount of clients attempting to access the database all still attempted to fix the invalid issue. This sent queries to the database which were all trying to restore the invalid value, however once they restored the value, it was still invalid since the persistent store version was invalid.

Creating a feedback loop which would not allow the databases to actually be fixed, without anything very quickly being replaced with the invalid value again reasoned a major decision from the network. Facebook had no choice but to actually turn the site off so that the databases may recover. Once they were recovered, the social networking site slowly let users access the service again, and they have since turned off the automated system to verify these configuration values. They are currently looking into new designs and models to prevent another outage such as this in the future.

Facebook said that this is the worst outage they have had in over four years, and wish to apologize for the issue and want you to know that they are very serious about the performance and reliability of the social network.

Report a problem with article
Previous Story

Update: Internet Explorer 9 doesn't require Windows 7 SP1

Next Story

Neowin looks at the #NewTwitter

68 Comments

Commenting is disabled on this article.

It's crazy the Facebook outage caused such frenzy. Like tons of other people, I've been using Facebook for several years now and I can't remember any other time when it was down for that long. Do you think Facebook's communication with users could have been better during this process? I think they should have sent out an email keeping everyone up-to-date on what was happening.

This wouldn't have been such an issue, except after the problem had been fixed, the numerous amount of clients attempting to access the database all still attempted to fix the invalid issue. This sent queries to the database which were all trying to restore the invalid value, however once they restored the value, it was still invalid since the persistent store version was invalid.

Creating a feedback loop which would not allow the databases to actually be fixed, without anything very quickly being replaced with the invalid value again reasoned a major decision from the network.

In English please?

This news only affects people who value Facebook more than oxygen. Would have been good for the gene pool if the outage was a little longer.

C_Guy said,
This news only affects people who value Facebook more than oxygen. Would have been good for the gene pool if the outage was a little longer.

+1

Saw this article yesterday, but saw it on Good Morning Amreica this morning talking about how everyone was freaking out!!

My gosh people, get a life!!

Wow, I'm amazed, how a defense system actually stood upon their own owners, this means they have the site well prepared, pity for people who actually got to see the sky, oh well.

Jose_49 said,
Wow, I'm amazed, how a defense system actually stood upon their own owners, this means they have the site well prepared, pity for people who actually got to see the sky, oh well.

Sky... Tell me of this thing you call a sky.

In other news, the Wall Street Journal reported that for an unknown reason, the economy & business productivity showed a minor uptick for 2.5 hours on Wednesday.

For that 2.5 hours, I had to say hi to the kids, pet the dog and make love to the wife. Now my world is back to normal thank god... Now family get back to your own lives.

Cellar Dweller said,
For that 2.5 hours, I had to say hi to the kids, pet the dog and make love to the wife. Now my world is back to normal thank god... Now family get back to your own lives.

Oh don't worry, tomorrow you will be gathering people to raise a barn or wack someone

If you're using Internet Explorer you can just type Facebook<Ctrl><Enter> if you really need to shave down the keystroke count.

A quote from my Facebook news feed I saw yesterday.

"Facebook was down an unusually long time today. Just wait, in nine months, the population will go up by 2%."

This is one of those scenarios like when the power/cable/internet goes out and you have to make your kids actually go outside and play with real people!

I thought I'd been spending too much time on Facebook lately, but I guess maybe I'm not that bad since I didn't even notice it was down. Meanwhile friends and family were apparently spazzing out.

Honestly, how on earth is this news. People dying in the world but no facebook has gone down hahah

I did notice it though and happened at the msot inconvient of times

mikemyres said,
Honestly, how on earth is this news. People dying in the world but no facebook has gone down hahah

I did notice it though and happened at the msot inconvient of times

People die every day, so THAT is not news. Facebook doesn't go down every day.

The outage, which lasted approximately 2.5 hours...

Facebook said that this is the worst outage they have had in over four years

Now it could be that they're talking about the worst outage tech-wise for them, but to me I think that comes across as a good track record.

Facebook must have been down for Windows users for that 2.5 hours as i had a Linux user not be able to acess it for under 2 minutes, hit refresh and boom it was back.

soldier1st said,
Facebook must have been down for Windows users for that 2.5 hours as i had a Linux user not be able to acess it for under 2 minutes, hit refresh and boom it was back.

Yes. Facebook is biased against Windows.

soldier1st said,
Facebook must have been down for Windows users for that 2.5 hours as i had a Linux user not be able to acess it for under 2 minutes, hit refresh and boom it was back.

Yeh Facebook obviously hates Windows, somehow a server side issue became client side

And lol at your sig:

Ubuntu Linux has overtaken Windows 7 as it is a very logical thing to do. it Masters where Windows 7 does not. Windows 7's blood is very thin and tinted but Ubuntu Linux's Blood is thick and strong to the bone as it was meant to be.

soldier1st said,
Facebook must have been down for Windows users for that 2.5 hours as i had a Linux user not be able to acess it for under 2 minutes, hit refresh and boom it was back.

feedback on the forums generally stated one of three scenarios: Facebook was completely down, Facebook was extremely slow, or Facebook was relatively normal.
I don't think the OS had anything to do with it, it was luck of the draw.

EDIT: Ah, Razorfold's finds have explained why your comment is so ridiculous.

soldier1st said,
Facebook must have been down for Windows users for that 2.5 hours as i had a Linux user not be able to acess it for under 2 minutes, hit refresh and boom it was back.

You're... trollin? I can't tell.

Uhuh. Tell me then, how did you manage to get on Facebook whilst they had turned off their servers? Some magical Linux voodoo? :L

~Johnny said,
Uhuh. Tell me then, how did you manage to get on Facebook whilst they had turned off their servers? Some magical Linux voodoo? :L

Of course, didnt u know Linux has special powers ?

~Johnny said,
Uhuh. Tell me then, how did you manage to get on Facebook whilst they had turned off their servers? Some magical Linux voodoo? :L

Yeah special trolling cache

soldier1st said,
Facebook must have been down for Windows users for that 2.5 hours as i had a Linux user not be able to acess it for under 2 minutes, hit refresh and boom it was back.

Most Active in: Windows Client Support (7, Vista, XP), trolling a lot?

Omg I can't farmville!!!!

Everyone at work was hounding me cause of this. I'm the network admin and they thought I was messing with them and their facebooking. What clowns. I'm not doing that till next week lol

Swerz said,
Omg I can't farmville!!!!

Everyone at work was hounding me cause of this. I'm the network admin and they thought I was messing with them and their facebooking. What clowns. I'm not doing that till next week lol

You're the network admin at work and you even allow the employees to access it? Must not get much done during the day!

Facebook, Twiiter, MySpace and any of those sites aren't allowed at my work place.
Thank goodness!

The person above who posted he "hanged" himself is another one of those misfits that can't even do that right!! LOL

cork1958 said,

You're the network admin at work and you even allow the employees to access it? Must not get much done during the day!

Facebook, Twiiter, MySpace and any of those sites aren't allowed at my work place.
Thank goodness!

The person above who posted he "hanged" himself is another one of those misfits that can't even do that right!! LOL

My company uses them for clients. if you block Social media in an advertising company your asking for LOTS of trouble and lost business.

Swerz said,
Omg I can't farmville!!!!

Everyone at work was hounding me cause of this. I'm the network admin and they thought I was messing with them and their facebooking. What clowns. I'm not doing that till next week lol

What kind of company do you work for? They let them play farmville o.O

Epic0range said,
My company uses them for clients. if you block Social media in an advertising company your asking for LOTS of trouble and lost business.

Likewise, massive companies that have lot's of departments tend to not block things like that because it's likely SOMEONE SOMEWHERE actually needs them as a tool. Conversely, since FB updated their game section to all reside in one sub-domain, my company has blocked all FB games.

timster said,
i hanged myself earlier this evening because of the lack of facebook

You did it wrong if you are posting

timster said,
i hanged myself earlier this evening because of the lack of facebook

John Edward, where are you? Dead people keep talking to me!!!!!!!!!!!!

dreamsburnred said,
2.5 hours isnt that bad really...even though its the busiest site in the world.

Windows update is busier than facebook.