Got an alert yesterday afternoon in my email that the 16TB RAID-5 Array in my file server at home had marked a drive offline and put itself in a degraded state. Looking at the physical disk list, Drive 2 was missing, and the logs contained an error that looked like there might be a cable problem. I shut the box down, and when I got home, checked all the cables to each drive. I powered the box back up, disk was there, and the array started rebuilding. Array finished rebuilding, Megaraid storage manager however produced a new error, something about "Puncturing BAD BLOCK"
During this time, I was able to access files fine on the array, so it's not gone. Regardless, I shut the box down again, swapped drive 2 out for a new 2TB drive, set the rebuild rate to 100%, and let it go this morning. It's still rebuilding, but I saw the following again in the log:
It's still rebuilding, and I'm going to let it finish. What does this "puncturing bad block" message mean? I've tried googling it, but from what I can find, it just looks like the controller found a bad block, and is either marking it as bad, or skipping it? I see that it shows ":5" meaning disk 5, as well as 2...does that mean theres a bad block on disk 5 that it copied to the new spare? I'm about to go out and buy a few more 2TB drives and replace possibly disk 5 as well... but wanted to see what you all thought.
Oh, and I did try and produce a backup immediately, but the backup tool failed due to a media error... Which is strange because I am able to copy files manually off the array. After it finishes rebuilding with the new drive I'm going to do a manual copy off to get a recent backup.
In case anyone is interested in drive information and system info, here are my system specs and history:
Samsung HD204UI drives running 24x7 for ~2 years
Dell PERC 6/I
Asrock Z75 Pro 3
Intel Xeon E3-1230 V2
While running 24x7 for ~2 years without issue, it looks as if the drives are reaching their limit.