Google restoring Gmail accounts from backup

There's something to be said about a good backup solution, and Google obviously take it seriously after posting that it has started to restore all the "lost" emails and settings from a backup, hours after users began complaining about their accounts being wiped of all data.

The company blamed a software bug for the incident and said just 0.02% of Gmail customers were affected, initially it had claimed as much as .29% were affected and then revised it down to .08% of its 170 million users.

"I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centres?" asked Ben Treynor, Google's site reliability czar, in the firm's official Gmail blog.

"Well, in some rare instances software bugs can affect several copies of the data. That's what happened here," he added.

He noted that Google backs up data on offline tapes, which are protected from software bugs. "But restoring data from them takes longer than transferring your requests to another data center, which is why it's taken us hours to get the e-mail back instead of milliseconds," he said.

"Thanks for bearing with us as we fix this, and sorry again for the scare," he concluded.

A detailed incident report has also been submitted to the Apps Status Dashboard, and Treynor also noted, "email sent to you between 6:00 PM PST on February 27 and 2:00 PM PST on February 28 was likely not delivered to your mailbox, and the senders would have received a notification that their messages weren’t delivered"

Google Docs and Calendar, were unaffected, but users will be happy to know that their data was carefully looked after.

Report a problem with article
Previous Story

IE9 reaches 36 Million downloads; Increases market share

Next Story

Microsoft to show off Windows 8 Tablet design demos in June

21 Comments

Commenting is disabled on this article.

what they wrote at this blog... http://gmailblog.blogspot.com/...back-soon-for-everyone.html

magine the sinking feeling of logging in to your Gmail account and finding it empty. That's what happened to 0.02% of Gmail users yesterday, and we're very sorry. The good news is that email was never lost and we've restored access for many of those affected. Though it may take longer than we originally expected, we're making good progress and things should be back to normal for everyone soon.

I know what some of you are thinking: how could this happen if we have multiple copies of your data, in multiple data centers? Well, in some rare instances software bugs can affect several copies of the data. That's what happened here. Some copies of mail were deleted, and we've been hard at work over the last 30 hours getting it back for the people affected by this issue.

To protect your information from these unusual bugs, we also back it up to tape. Since the tapes are offline, they're protected from such software bugs. But restoring data from them also takes longer than transferring your requests to another data center, which is why it's taken us hours to get the email back instead of milliseconds.

Read more at the link above...

Tape backups are quite quite common. In fact Iron Mountain (the company) specializes in record storage on behalf of other companies -- digital ones are on magnetic tapes.

Manksgloob said,
Tape backups are quite quite common. In fact Iron Mountain (the company) specializes in record storage on behalf of other companies -- digital ones are on magnetic tapes.

They do? Good to know

Manksgloob said,
Tape backups are quite quite common. In fact Iron Mountain (the company) specializes in record storage on behalf of other companies -- digital ones are on magnetic tapes.

They do? Good to know

Manksgloob said,
Tape backups are quite quite common. In fact Iron Mountain (the company) specializes in record storage on behalf of other companies -- digital ones are on magnetic tapes.

we used them for 2 years, horribly overpriced for what they do, now we use Business record management, which is half the cost for the same thing

Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

I'm not sure, but I occasionally hear tapes being referred to when people are talking about DVDs or external drives.
I think in this article, "tapes" are referring to the swarms of hard drive hubs they would need to backup the email data.
(Yes, I do know about the existence of data tapes)

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

Tape backup is still widely used, it is one of the best solutions.
And what others ways are there for backup? on other harddisks, they fail very often. on DVD's, immagine how many they will need

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

They work and are reliable. I worked at a client who was using tape backups for a long time and they were reliable. Never had loss of data.

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

I don't know a whole lot about the reliability but tape back ups seem fairly common.

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

Banks have used them foe decades, gonna go with very reliable

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

"Reliability
Gartner Group estimated that 10 to 50 percent of all tape restores fail. Storage Magazine and Gatner reported that 34% of surveyed companies never test a restore from tape, and of those that do test, 77% experienced tape backup failures."

Assuming that has improved in recent times.

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

i used to backup using a tape and it worked pretty well until i needed to restore a backup.. it failed and i was unable to do a partial restore. Solution :dvd backup, cheaper, standard and can be restored without extra hardware.

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

Every network i've ever worked on has used some kind of tape backups. Nowadays it's usually disk-to-disk-to-tape, or disk-to-disk-to-disk-to-tape or something where the data is stored on one or more backup SANs keeping the data readily available, but you'll find at the end of the chain it's almost always stored offline to tape and then shifted off-site.

Tapes offer huge amounts of storage compared to other media. For example the commonly used LTO4 will hold 1.6TB on one tape. Where I work, we store weekly/monthly/yearly backups off site, and the yearly ones have been there for many many years. As long as the storage is temperature/humidity controlled, they stay good for years and years.

dancedar said,
Tape backups? Interesting. Anyone know the reliability of this, is it 100% after several years of storage? Not doubting Google's choice of backup, just wondering on the tech side.

You don't rely on 1 tape to last for years... whoever uses tape as backup would periodically backup on other tapes after a month or week or so... so tapes that are a year old are irrelevant if it fails as long as there are other tapes containing more recent backups.

lots of us use tape backups...... we have a tape for every day for our SQL servers and a tape for our file systems and a tape for all the servers.... so in a remote location we have over 1,000 tapes sitting locked up in a vault... they are a good backup, we rarely have failures restoring them... but then we dont overwrite them

It is indeed a relief to know that at least some company accepts the responsibility they are morally obligated to when it comes to user data. In this fast approaching cloud computing world, everyone should be well aware of which providers care about it's user's data and which don't.

Xenosion said,
It is indeed a relief to know that at least some company accepts the responsibility they are morally obligated to when it comes to user data. In this fast approaching cloud computing world, everyone should be well aware of which providers care about it's user's data and which don't.

+1