PERC 6/I "Puncturing bad block" ?

[SirEvan]

Got an alert yesterday afternoon in my email that the 16TB RAID-5 Array in my file server at home had marked a drive offline and put itself in a degraded state. Looking at the physical disk list, Drive 2 was missing, and the logs contained an error that looked like there might be a cable problem. I shut the box down, and when I got home, checked all the cables to each drive. I powered the box back up, disk was there, and the array started rebuilding. Array finished rebuilding, Megaraid storage manager however produced a new error, something about "Puncturing BAD BLOCK"

During this time, I was able to access files fine on the array, so it's not gone. Regardless, I shut the box down again, swapped drive 2 out for a new 2TB drive, set the rebuild rate to 100%, and let it go this morning. It's still rebuilding, but I saw the following again in the log:

It's still rebuilding, and I'm going to let it finish. What does this "puncturing bad block" message mean? I've tried googling it, but from what I can find, it just looks like the controller found a bad block, and is either marking it as bad, or skipping it? I see that it shows ":5" meaning disk 5, as well as 2...does that mean theres a bad block on disk 5 that it copied to the new spare? I'm about to go out and buy a few more 2TB drives and replace possibly disk 5 as well... but wanted to see what you all thought.

Oh, and I did try and produce a backup immediately, but the backup tool failed due to a media error... Which is strange because I am able to copy files manually off the array. After it finishes rebuilding with the new drive I'm going to do a manual copy off to get a recent backup.

In case anyone is interested in drive information and system info, here are my system specs and history:

Drives: Samsung HD204UI drives running 24x7 for ~2 years

RAID controller: Dell PERC 6/I

Motherboard: Asrock Z75 Pro 3

Processor: Intel Xeon E3-1230 V2

While running 24x7 for ~2 years without issue, it looks as if the drives are reaching their limit.

Thanks!

[SirEvan]

Here's the current status of the rebuild...50% done almost. Ignore the "sense" errors, for some reason dell controllers seem to do that when you click a disk in megaraid storage manager

[SirEvan]

hmm stil getting errors with the 2nd and 5th disks. trying to perform a backup again now..if this doesn't work I'm going to have to do a teracopy copy of everything off the array and blow it away.

[Martog]

Run a consistency check on it and see if it goes away.

If it does not, here is what dell says about it (We've been hearing about punctured arrays this year when we replace customer's drives that keep dropping and coming back online) and assumes you have Dell Openmanage server administrator installed at the step with C:\Program Files\Dell\SysMgt\oma\bin\omconfig.exe.

Here is the procedure to try and recover from the punctured stripe:

The best way to fix this is to delete the array (at the controller level), recreate it, perform a full initialization, and then restore the data. This will fix the problem 100% of the time.

The alternate procedure is to:

1. Turn off Patrol Read
   
2. Perform a full system backup with states
   
3. Once the backup is completed, look in the backup job log for missing or corrupted files.
   
4. Got to the path where it is annotated in the job log (ie, c:\windows\...) and either delete the file or restore from the backup. If the file is essential, then you will need to restore it.
   

Once this is done, run a consistency check:

1. Click Start, point to Programs, Accessories, System Tools, and choose Scheduled Tasks.
   
2. Double-click Add Scheduled Task.
   
3. Click Next.
   
4. Click Browse... and locate the file omconfig.exe. The default location of this file is
   C:\Program Files\Dell\SysMgt\oma\bin\omconfig.exe
   
5. Click Next.
   
6. Enter a task name. This is cosmetic and serves as a reminder to you of the scheduled task's purpose.
   
7. Select how often you want the task to run, and then click Next.
   
8. Choose the date, time, and month options, and then click Next.
   
9. Supply the user name and password of the account that will run the scheduled task.
   
10. Click Next, and then click Finish
    
11. Run a CHKDSK
    
12. In about a week turn on Patrol Read.
    

NOTE: if this procedure fails, you are still looking at wiping out the drive and rebuilding it.

You'll probably have to blow the array away at the controller level.

[SirEvan]

Run a consistency check on it and see if it goes away.

If it does not, here is what dell says about it (We've been hearing about punctured arrays this year when we replace customer's drives that keep dropping and coming back online) and assumes you have Dell Openmanage server administrator installed at the step with C:\Program Files\Dell\SysMgt\oma\bin\omconfig.exe.

You'll probably have to blow the array away at the controller level.

interesting find. Thanks for that. I had an extra drive laying around, so i tossed it in for Drive 2, booted into the raid controller tool, and let it rebuild. once done, booted into windows and ran a consistancy check. consistany check failed, same message but on drive 5...same block. I tried to run a backup, but the backup fails due to a media error (MSM shows an error as well). I've pulled drive 5, and am in the process of running samsungs extended drive test on it, but from what I can see from smart and the quick test, the drive looks fine, so I'm wondering if the controller might have an issue.

More info:

Controller has a good battery backup on it, and server is plugged into a 1 hour UPS.

I wiped a drive and placed it in drive 5's spot, array is rebuilding right now in dos. Once it's 100%, I'm going to boot again and try a backup. If the backup fails, I'm going to resort to using teracopy or something to just do a mass copy to my backup array, then blow the array away.

FWIW, I recently removed full drive encrpytion from the array, and while I was in the process, changed the stripe size from 512KB to 64KB (for faster access to smaller files). Before doing this, the array was just fine for over 19000 hours (800 days), and in the span of about 7 days, I've now got errors...perhaps Consumer drives aren't liking 64KB stripe sizes (more reading/writing to the drives I assume?)

Thanks. I'll give that a shot tomorrow