48 posts in this topic

(Thread dedicated to photo-14624.gif)

Hello,

Before I get started, please lets not let our knowledge get mixed with emotions because someone perfers to do this or that a certain way....

Also, I do want to make clear this from the start: RAID is NOT a backup solution. It is not its purpose and it is indeed important having (both) offline (completely offline, not connected to any ouside source and/or network) and online backups.

This spoiler contains info about RAID. Skip it if its not of intrest or you know how it works:

               

As some background to RAID itself:

There are "7 standard" RAID levels (resume):

0 - Splits data evenly across two or more disks (striped) without parity information for speed.

1 - An exact copy (or mirror) of a set of data on two disks.

2 - Stripes data at the bit (rather than block) level, with error correction (not really used)

3 - Uses byte-level striping with a dedicated parity disk. (Havent heard much usage either)

4 - Block-level striping with a dedicated parity disk

5 - Block-level striping with parity data distributed across all member disks

6 - Extends RAID 5 by adding an additional parity block; thus it uses block-level striping with two parity blocks distributed across all member disks.

If you wish to read a bit more: http://en.wikipedia.org/wiki/Parity_bit#RAID

The reason I said "7 standard" is because RAID0 is not RAID as it has no parity infomation.

Most RAID controllers support 0, 1, 5 and 6.

Having said that, Ive been using RAID5 for a few years. It has saved me quite a few times when a drive has failed. Its read/write performance is not best (more so with a slow RAID card) but it gets its purpose done.

Now, the guru Ive dedicated this thread to (and a couple of others) have suggested using different RAID levels. I want one that gives me the same as RAID5 but better (that doesnt even make any sense but it just came out like that).

Let (peacefully) debate this :)

Thanks

Share this post


Link to post
Share on other sites

Already went over some of them in your other thread..

 

Why do we need to repeat ourselves?

 

I am more interested in your statement "It has saved me quite a few times when a drive has failed."  After you start out your topic clearly in agreement that raid is not a black up solution.

 

So what did raid save you from?  Not having to restore from backup?  If you were using just a normal drive pool, my critical files are on multiple disks in the pool - and only thing that would had to been restored from backup is the stuff on that disk that I deemed non critical in the first place.

 

As I stated in your other thread what portion of your data is deemed "critical" it is not always cost effective or performance wise to keep parity on data that is not critical.

 

The other aspect where raid fails is your spending $$ to for 9TB of storage - when you currently have 1TB?  Unless your going to ramp up need very very quickly you have $ sitting there wasted using elec, just waiting to fail ;)

 

So myself and others have already gone over some of the other options - how about you tell us why those options don't sit well with you, other than your legacy attachment to a old technology.

Share this post


Link to post
Share on other sites

I use a mixture of RAID 1 and 5. 1 for OS, and files not accessed very often and I use RAID 5 for file storage mainly due to only dropping ~33% storage. has nothing to do with added protection as both can only recover from a single drive failure.

 

Example. I am putting together a proposal (in its initial stages) for 2 servers to be used for VM and archived data. I am looking at HP DL380's with 8 3.5" drive bays. I plan to use RAID1 for OS, iso's and similar files that will be rarely used and then a 3 drive RAID 5 with either 3 or 4TB drives for the VMs and the archive data leaving 3 drives spare. By the time this RAID 5 has been used up I will be able to buy larger capacity drives for less and create a new RAID 5 with larger capacity than the original and I expect to be replacing the server before if fill that space up.

 

We are only a small office hence doing things like this. We backup to the cloud and are also planning on creating a local backup for a server failure or similar. If it was a larger enterprise I would be looking at clustering with SANs and similar technology.

Share this post


Link to post
Share on other sites

RAID 1, RAID10, or RAID6 is what I typically recommend.

 

RAID-5 has a very high chance of a second disk failing during rebuild on large disk arrays with high capacity 2TB+ disks rendering all of your data lost.

 

RAID-1 for Boot or mostly sequential data that you want ot protect.

 

RAID-10 or RAID-6 for everything else.

 

RAID-10 has better performance and better protection from disk failures and faster recovery since its just a straight copy operation than RAID-6 but you lose 1/2 of your capacity.

 

RAID-6 is a decent compromise capable of surviving (2) failed disks.

Share this post


Link to post
Share on other sites

I use a mixture in my server setup at work also.

 

OS  = RAID1 (2 x 136GB SAS 15,000RPM)

Data = RAID10 (4 x 300GB SAS 15,000 RPM)

 

I'll hopefully move those 300GB drives to the OS RAID and then upgrade the RAID10 to 4x1TB drives. :)

 

As to why - The OS is images and backed up onsite and offsite in the event of fire or sprinkler malfunction! (GRRRRR :p ) For just a simple HDD crash however, we can keep running till the drive gets replaced.

 

The data drive - That has some VHDs on it so needed faster storage but also wanted redundancy in the event of HDD failures. RAID10 is faster than RAID5 for WRITES so went that route instead of RAID5. It has the side benefit of allowing 2 drives to fail (only if separate arrays - so one of each in my case of 4 drives) but I have drives on standby just in case one fail I can replace it before the others start to.

Share this post


Link to post
Share on other sites

I'm not sure what there is to debate? All the different RAID levels have their pros and cons, some with more cons than others but it's all common information.

Share this post


Link to post
Share on other sites

.....

 

RAID-5 has a very high chance of a second disk failing during rebuild on large disk arrays with high capacity 2TB+ disks rendering all of your data lost.

.....

 

This had completely slipped my mind! still a simple change for my server set up. 2x RAID 1. One for VM the other for the archive data perhaps....

Share this post


Link to post
Share on other sites

This had completely slipped my mind! still a simple change for my server set up. 2x RAID 1. One for VM the other for the archive data perhaps....

You should just stay away from RAID5 for the most part. Three HDDs in a RAID5 barely gives you any performance improvement over a single or mirror and you just complicate the process with parity adding overhead. RAID10 is much better with I/O and is what you should be using for heavy use.

 

Pretty much the only scenarios I use RAID5 are with SSDs since the performance is so staggering or with small HDD arrays for backups but if they get more than about 6 drives, I'll use RAID6.

Share this post


Link to post
Share on other sites

You should just stay away from RAID5 for the most part. Three HDDs in a RAID5 barely gives you any performance improvement over a single or mirror and you just complicate the process with parity adding overhead. RAID10 is much better with I/O and is what you should be using for heavy use.

 

Pretty much the only scenarios I use RAID5 are with SSDs since the performance is so staggering or with small HDD arrays for backups but if they get more than about 6 drives, I'll use RAID6.

 

This server chassis comes in either 8 3.5" bays or if I remember correctly 16 2.5" bays so I will be having another think about the setup when I take another look at this project which will be very soon. may take a look at RAID1 with 2 SSD's for OS then RAID10 with the largest drives I can find for the system leaving 10 bays to expand with. Food for thought though.

Share this post


Link to post
Share on other sites

I believe this discussion is more related to home or small office - not an enterprise setup. This is the track I took in the other thread. Where he was asking for a 4 bay nas, neither model he was looking at would be used in an enterprise.

So I look at this from point of view of type of files I have in my home, and what I serve up off my storage.. These are media files, video and music mostly. All of which have no need of parity, since if they are lost I can replace them off their original media sitting on my self or if need be gotten again via other channels ;)

Now what is critical is a small subset of these files, my home video for example. These I have backed up in multiple locations on different storage, cloud, other disk in different system, optical on my self and another copy at my son's home, etc. So why should I create parity for say my rip of scarface or my grateful dead cds? Now for piece of mind - these "critical" files are also duplicated onto another disk in the pool automatically, so you get the same sort of protection you get with raid 1, while only using a subset of your storage pool for these non replaceable "home movies".

Money spent on that parity seems wasted to me, if that drive died where those files are stored I could just rerip (replace from my backup). There is no critical need for these files to be online in case of disk failure. Which is what the purpose of raid - this has little use in home setup or where only a subset of files in the storage is considered of a critical nature and needs to be online even with hardware failure such as a disk.

In his example of 1TB of storage - why would raid 1 not be better option? He uses 2x3TB or even 2x2TB and he covers his storage needs at lower cost while still having room for growth that should cover him for quite a bit of time.

Raid 5 is better suited when you need specific amount of storage but can not achieve this within specific cost constraints with a mirrored setup, Say he needed 6TB of storage - well there is not 6TB disks as of yet.. So he could do say 4x3TB in raid 10 or 5, or he could do 3x3TB in raid 5, or 4x2TB in Raid 5, etc. But again what amount of that storage requires parity? All of it then sure raid 10 or 5 might make sense.

Or what if he has only 1 TB of critical and 5TB of stuff that is nice to have digital access to - like movies and music. I could accomplish that with 2x4TB in a pool where my 1TB is duplicated on each disk. And this leaves me 6TB of storage - 5 of it for my other stuff and 1TB of growth. At a much lower cost and better flexibility. Since I only need 2 disks. And such time that I need more space I could add another disk to the pool - and its connection and size is no matter, it could be say a 2TB esata or usb even, now if I wanted I cold duplicate my 1TB of critical to all 3 disks in the pool and still have an extra 1TB to play with. Lots of different scenarios viable in the growth of my storage pool.

Not having to put min 3 disks into use all at the same time, allow me to grow my storage using size and connection type that gives me best bang for the buck. As we all know, disks only get bigger, faster and cheaper next month. Such a methodology allows me to stagger disks purchase to take advantages of lower cost when I actually need the storage, not having to calculate how much I need to put online now to have what I need 2 years from now.

This can allow for retirement of your OLD disks before they fail as you just naturally grow your storage replacing older/slower/smaller disks with faster/bigger ones while not requiring more slots.

If need be I can move these disks in my pool to new box - not having to worry about the raid controller in it, or lack of one. Say I need to take a bunch of media to a remote location - I can just take the disk out of the pool and access the files directly via anything that can read the filesystem I used - in my case just common ntfs.

The software I am using monitors the disks, and can notify me of possible issues be it physical issues reading sectors on the disks or smart information pointing to possible failure, etc. It can even move files off those disks in the pool if space is available in the pool.

Lets take a look at your 4x3TB - from the math I have seen, there is something like a 56% chance that with reading 10TB of data that you will encounter an unreadable bit and your rebuild will fail. So when your 1 disks fails its a coin toss if your going to be able to rebuild the array from that parity you spent good money on creating. Also you more than likely built that array from disks purchased all at the same time, most likely in the same batch - once 1 disks fails in a batch, the probability of another disk failing in that same batch increases, etc.

What sort of disks are you using to create this raid in the first place - are they enterprise quality designed to be in a array where they are read and written too constantly? Those disks are normally more costly, does this added cost make sense in a home setup to serve media files?

Its great you have had great success with raid 5 in the past, does not mean it meets the needs of today or makes sense with the size and speed of disks that are available today and the other ways to merge them together to so that their combined space is accessible in one location.

Not talking enterprise where files need to be online 24/7/365 or money is lost.. Talking a home or small office, etc.. Small budgets, etc. Even then your seeing the enterprise move away from your typical raid arrays as as these disks get larger and larger the likely hood of failure on a rebuild grows.. From the math I have seen if your talking 100TB is like 99.9% sure your going to hit a unreadable bit trying to rebuild the array on a disk failure.

edit: BTW for anyone curious I am using https://stablebit.com/ and can not say enough about their support.. It just freaking rocks!! For the small cost of their software, you can not find better support. I recently ran into some issues using a 3TB 4k sector disk in my n40l where I use passthru or physical RDM to give access to the disk to the VM. So that it can read smart, etc. The windows vm just was not seeing the full size of the disk or the gpt information correctly, etc. Now the esxi saw it as 3TB no problem, and could manipulate partitions on it just fine, using partedutil, etc. But windows was reporting it as -128MB size or 0 in Disk Manager. If I connected it to a linux VM had no problems using gdisk to manipulate and verify the gpt, and parted to create partitions, etc. So I just created the 3TB partition in linux and then attached to my windows VM.. Working great - but the scanner portion of their software was just using the info windows was giving about the disk.. Which was not correct.. So in a few days chatting with them via their support system the developer created some new beta versions of the scanner software that looked to the disk directly for info when windows was reporting odd information.. Works great now, scanner and pool both report correct size of the disk, scanning of all the sectors works, etc. etc. They even offered to remote in and take a look if need be to get their software to work even clearly the issue is windows and or esxi passhtru, etc. In the long run no need for this - but did have the meeting scheduled, etc. I can not really say enough good things about their product and support.

1 person likes this

Share this post


Link to post
Share on other sites

I work in the support department for a large company and I typically deal with all the RAID issues that come around.

 

I want to first start out by saying the performance hit with RAID 5/6 isn't that much of a factor anymore with the speed and resources of modern day RAID controllers.

 

RAID 10 is good but if you don't maintain it, it has a very big weakness. People like to tout how it can survive half the drives failing, however they always neglect to mention that it can only survive 1 drive from each RAID 1 leg. I've seen too many times when a drive from a RAID 1 leg fails but a replacement can't rebuild due to errors on the other member. When that happens nothing you can do other than start that array over from scratch.

 

RAID 60 is a good solution if you want to go with multiple drives, but, I would keep the amount of drives limited to about 12(and with hotspares, have to have hotspares). Too many drives just basically increases the chances for complete failure exponentially.

 

However, I prefer doing a RAID setup on ZFS to hardware RAID. I used to hate software RAID (even still do, other than RAID Z 2/3)

Share this post


Link to post
Share on other sites

RAID should be used to increase storage performance, and to decrease downtime caused by hardware failure. You should also be using a high-quality dedicated controller for RAID. For any RAID setup, you should be doing automatic disk and array verification, as well as backing up anything you cannot bear to lose to another location.

 

On my workstation I use a RAID-0 configuration, where all the critical data is backed up to a network location. The purpose is to combine storage into one volume, and to increase storage performance.

 

On my home server I use two 4-disk RAID-5 arrays (Though I recommend RAID-10 to everyone else). The reason being, using two 5-in-3 SATA bays, I have the capacity for 10 disks. I prefer the capacity for increased storage to the benefits of RAID-10. I backup what is important to an external disk, and have only myself to blame should there be a catastrophic failure.

 

In my professional experience, hot-spares are a bad idea. You end up with depending on a disk that is as old as the one failing to sustain a trouble-free array rebuild of a typically large amount of data being sequentially written. In one case, I was evaluating a company's SBS server, and I decided to do a disk integrity verification task on their RAID-5 hot-spare. It failed. IMO, keeping a pre-verifed spare on-hand and having the array status notifications properly configured is better.

1 person likes this

Share this post


Link to post
Share on other sites

I use mirrored right now, maximal redundancy.

Share this post


Link to post
Share on other sites

Hello :)

Already went over some of them in your other thread..

Why do we need to repeat ourselves?

I am more interested in your statement "It has saved me quite a few times when a drive has failed." After you start out your topic clearly in agreement that raid is not a black up solution.

Well, it has saved in the sense that a disk fails and I do not lose any data. Once a disk fails, my procedure is the following (and i know most of you arent going to like this).

1: Make sure important backups are up to date (all of the failures Ive had with one HDD, all the backups were extremely up to date so no worries there)

2: Send off the HDD to get a replacement

3: Do not use the PC at all.

Now number 3 must KILL some of you inside saying "Double you tee ef! What are you going without your PC for at least a week?". The truth is that most of that RAID has "replacable data". There isnt anything (other than important backups which I always keep up to data) life changing. All I will waste is time, gaining it all back. Besides, its nice also to disconnect a bit from the PC and enjoy other aspects of life.

So what did raid save you from? Not having to restore from backup? If you were using just a normal drive pool, my critical files are on multiple disks in the pool - and only thing that would had to been restored from backup is the stuff on that disk that I deemed non critical in the first place.

As I stated in your other thread what portion of your data is deemed "critical" it is not always cost effective or performance wise to keep parity on data that is not critical.

The other aspect where raid fails is your spending $$ to for 9TB of storage - when you currently have 1TB? Unless your going to ramp up need very very quickly you have $ sitting there wasted using elec, just waiting to fail ;)

So myself and others have already gone over some of the other options - how about you tell us why those options don't sit well with you, other than your legacy attachment to a old technology.

Well, from a certain point of view, it really saved me from nothing. Time is wasted. Efficiency is nulled as I dont use my PC (I do enjoy the break so its 50% purposly and another 50% not). I guess it comes down to pure lazyness and the ability to plug in a new drive and say "F it" and it does it all by itself without hardly intervention. That might not be the best way to do things but...

Those 9TBs are really, as I mentioned, future proofing. When I got them, I was working with less than 1GB files. Since now I have room to "move around" I working with 5GB+ files. I dont feel the need to watch spaces. Its something I REALLY hated in the past. When I needed new space, I would have to make a 2 hour trip or wait a week for a new hard drive to arrive. Personally, really got to me.

I'm not sure what there is to debate? All the different RAID levels have their pros and cons, some with more cons than others but it's all common information.

Thats what the thread is about. List the pros/cons in situations myself and other dont see...

I believe this discussion is more related to home or small office - not an enterprise setup. This is the track I took in the other thread. Where he was asking for a 4 bay nas, neither model he was looking at would be used in an enterprise.

Yes, this is SOHO. I would have other ideas in implement in a enterprise.

So I look at this from point of view of type of files I have in my home, and what I serve up off my storage.. These are media files, video and music mostly. All of which have no need of parity, since if they are lost I can replace them off their original media sitting on my self or if need be gotten again via other channels ;)

Now what is critical is a small subset of these files, my home video for example. These I have backed up in multiple locations on different storage, cloud, other disk in different system, optical on my self and another copy at my son's home, etc. So why should I create parity for say my rip of scarface or my grateful dead cds? Now for piece of mind - these "critical" files are also duplicated onto another disk in the pool automatically, so you get the same sort of protection you get with raid 1, while only using a subset of your storage pool for these non replaceable "home movies".

Money spent on that parity seems wasted to me, if that drive died where those files are stored I could just rerip (replace from my backup). There is no critical need for these files to be online in case of disk failure. Which is what the purpose of raid - this has little use in home setup or where only a subset of files in the storage is considered of a critical nature and needs to be online even with hardware failure such as a disk.

To me I rather waste money on parity of replacable data than have to go out and buy hardware 2+ hours plus away just to replace my storage. Maybe Im failing to realize something obvious....

In his example of 1TB of storage - why would raid 1 not be better option? He uses 2x3TB or even 2x2TB and he covers his storage needs at lower cost while still having room for growth that should cover him for quite a bit of time.

I believe its a pure act of laziness on my part: Instead of having 1TB of storage and when I run out of room (at lets say I have another HDD already) just plug it in, why not have it plugged in already waiting. Now you brought up a good point that is electricity/wearandtear not being used....

Raid 5 is better suited when you need specific amount of storage but can not achieve this within specific cost constraints with a mirrored setup, Say he needed 6TB of storage - well there is not 6TB disks as of yet.. So he could do say 4x3TB in raid 10 or 5, or he could do 3x3TB in raid 5, or 4x2TB in Raid 5, etc. But again what amount of that storage requires parity? All of it then sure raid 10 or 5 might make sense.

Or what if he has only 1 TB of critical and 5TB of stuff that is nice to have digital access to - like movies and music. I could accomplish that with 2x4TB in a pool where my 1TB is duplicated on each disk. And this leaves me 6TB of storage - 5 of it for my other stuff and 1TB of growth. At a much lower cost and better flexibility. Since I only need 2 disks. And such time that I need more space I could add another disk to the pool - and its connection and size is no matter, it could be say a 2TB esata or usb even, now if I wanted I cold duplicate my 1TB of critical to all 3 disks in the pool and still have an extra 1TB to play with. Lots of different scenarios viable in the growth of my storage pool.

Not having to put min 3 disks into use all at the same time, allow me to grow my storage using size and connection type that gives me best bang for the buck. As we all know, disks only get bigger, faster and cheaper next month. Such a methodology allows me to stagger disks purchase to take advantages of lower cost when I actually need the storage, not having to calculate how much I need to put online now to have what I need 2 years from now.

This can allow for retirement of your OLD disks before they fail as you just naturally grow your storage replacing older/slower/smaller disks with faster/bigger ones while not requiring more slots.

If need be I can move these disks in my pool to new box - not having to worry about the raid controller in it, or lack of one. Say I need to take a bunch of media to a remote location - I can just take the disk out of the pool and access the files directly via anything that can read the filesystem I used - in my case just common ntfs.

I want to confirm because I might not understand the correct terminology of what you said: What do you mean when you say that you put partiions of your HDDs in a pool?

For the record, I like my storage reprented by ONE drive letter. I dont like splitting partiions and or have each drive represent different things (videos, music, etc). Thats a personal prefrences.

Lets take a look at your 4x3TB - from the math I have seen, there is something like a 56% chance that with reading 10TB of data that you will encounter an unreadable bit and your rebuild will fail. So when your 1 disks fails its a coin toss if your going to be able to rebuild the array from that parity you spent good money on creating. Also you more than likely built that array from disks purchased all at the same time, most likely in the same batch - once 1 disks fails in a batch, the probability of another disk failing in that same batch increases, etc.

What sort of disks are you using to create this raid in the first place - are they enterprise quality designed to be in a array where they are read and written too constantly? Those disks are normally more costly, does this added cost make sense in a home setup to serve media files?

I believe I remember that I bought from at different dates or/and batches. I cant confirm that either.

In my (I guess luck) one disk has failed every 2-3 years. Some are there from the start, another failed 2 months ago or so and got quickly replaced.

One of my mistakes was indeed not buying enterprise quality HDDs.

Its great you have had great success with raid 5 in the past, does not mean it meets the needs of today or makes sense with the size and speed of disks that are available today and the other ways to merge them together to so that their combined space is accessible in one location.

Well, if its changable and makes sense I would change my RAID5 to something else.

Thank you BudMan and others.

Share this post


Link to post
Share on other sites

"I want to confirm because I might not understand the correct terminology of what you said: What do you mean when you say that you put partiions of your HDDs in a pool?"

My disks look like 1 drive to the system.. Lets call it H: just like a raid array is 1 disk to the system.

Where yours is made up of 3 disks of the same size and speed, my pool can be made up of multiple sized disks and or connection types. Currently it is a 2TB and a 3TB, it use to be 2TB with 750GB and another 750GB, giving me 3.5 total (just using the actual size of disks from the maker - not the actual useable space that you get with disks) For example 3TB is only 2.7TB..

What is nice is I can actually access each disk in the pool directly while the OS also sees it as part of the pool

post-14624-0-58376400-1381169189.jpg

So if you notice rdm2 is currently not part of the pool, since this is one of the older 750GB disks that was showing signs of possible failure. But I can directly access E or F, but their total size is presented to the OS as H..

post-14624-0-64403100-1381169357.jpg

So as you can see here

post-14624-0-93143700-1381169987.jpg

My molly folder in the pool is on 2 disks, if I had say 5 disks in the pool I could tell the software to keep everything in the molly folder on all 5 disks, or 3 of them if so desired. Or I can even take that down to the file level and say hey movie.mp4 make sure you keep that on 3 disks at all times.

This gives me my real time replication of my "critical" data while not having to worry about creating parity for every single file I have in the pool.. Like my neil diamond CD's -- which I could just rerip if disk failed that was storing those.

Share this post


Link to post
Share on other sites

It's all about RAID-Z.. as the Wikipedia article says:

 


RAID-Z avoids the RAID 5 "write hole" using copy-on-write: rather than overwriting data, it writes to a new location and then automatically overwrites the pointer to the old data.[10] It avoids the need for read-modify-write operations for small writes by only ever performing full-stripe writes. Small blocks are mirrored instead of parity protected, which is possible because the file system is aware of the underlying storage structure and can allocate extra space if necessary. RAID-Z2 doubles the parity structure to achieve results similar to RAID 6: the ability to sustain up to two drive failures without losing data.[11] In July 2009, triple-parity RAID-Z3 was added to provide increased redundancy due to the extended resilvertimes of multi-terabyte disks.[12]

Share this post


Link to post
Share on other sites

Hello,

"I want to confirm because I might not understand the correct terminology of what you said: What do you mean when you say that you put partiions of your HDDs in a pool?"

My disks look like 1 drive to the system.. Lets call it H: just like a raid array is 1 disk to the system.

Where yours is made up of 3 disks of the same size and speed, my pool can be made up of multiple sized disks and or connection types. Currently it is a 2TB and a 3TB, it use to be 2TB with 750GB and another 750GB, giving me 3.5 total (just using the actual size of disks from the maker - not the actual useable space that you get with disks) For example 3TB is only 2.7TB..

What is nice is I can actually access each disk in the pool directly while the OS also sees it as part of the pool

attachicon.gifpool.jpg

So if you notice rdm2 is currently not part of the pool, since this is one of the older 750GB disks that was showing signs of possible failure. But I can directly access E or F, but their total size is presented to the OS as H..

attachicon.gifdisks.jpg

So as you can see here

attachicon.gifduplicated.jpg

My molly folder in the pool is on 2 disks, if I had say 5 disks in the pool I could tell the software to keep everything in the molly folder on all 5 disks, or 3 of them if so desired. Or I can even take that down to the file level and say hey movie.mp4 make sure you keep that on 3 disks at all times.

This gives me my real time replication of my "critical" data while not having to worry about creating parity for every single file I have in the pool.. Like my neil diamond CD's -- which I could just rerip if disk failed that was storing those.

This concept seems sort ok like symbolic links to me in unix. Is this similar?

You have no parity at all; Meaning if a drive fails, you simply replace it and restore its data from a backup, right?

The hours I spend on Wikipedia and still some concepts are new, thanks to BudMan.

 

It's all about RAID-Z.. as the Wikipedia article says:

Well RAID-Z isnt standard so it isnt supported in most hardware dedicated towards RAIDs because I believe its actually a file system feature found in ZFS.

Good advice though.

Share this post


Link to post
Share on other sites

There is no parity - but you can have replication of your critical files (think raid 1)  Or even duplication to multiple disks.

 

As to the tech that makes it work -- not exactly sure.  You could somewhat simulate sim sort of setup with links sure.   But there is much more too it than that..

Share this post


Link to post
Share on other sites

Hello,

There is no parity - but you can have replication of your critical files (think raid 1)  Or even duplication to multiple disks.

 

As to the tech that makes it work -- not exactly sure.  You could somewhat simulate sim sort of setup with links sure.   But there is much more too it than that..

Is there a way to implement this pool in a VM? Just to see how it works and how to manage it?

Thanks

Share this post


Link to post
Share on other sites

Sure --- don't see how it would see the difference with a real disk a virtual one.  I am running in a vm currently, with the disks just passedthru to the vm.

Share this post


Link to post
Share on other sites

Well RAID-Z isnt standard so it isnt supported in most hardware dedicated towards RAIDs because I believe its actually a file system feature found in ZFS.

Good advice though.

 

Neither are other approaches such as DriveBender, DrivePool, Windows Storage Spaces, UnRAID etc.

I thought that this thread was about why 'standard' RAID levels were outdated and somewhat useless in the modern age.

Share this post


Link to post
Share on other sites

I am not exactly sure on the point of this thread either.. I tried to ask why he was so zero'd in on raid 5 for his small network setup when seems to me there are more cost effective and more flexible ways to access his storage in his other thread where he was looking for nas that supported raid 5.

He seems bent on spending money for no reason if you ask me - from the other thread it seems he has 1TB of data.. If he hell bent on parity/redundancy for his 1TB of data than a 2X2TB raid 1 would be better suited than a 4x3TB raid 5.. That is just a lot of space for no reason. Unless he is going to ramp up to close to 9TB of used space in say the next 6 months or so he is paying for spindles to spin for no reason.

I prob could of gotten by with adding only a 2TB disk, only reason I went with 3 was got a good price on it - and it only a few dollars more than the 2TB were going for. Got it for 109 when ST2000DM001 (2TB version is $99)

But even at that price he talking $440 for his raid 5 vs $200 for his raid 1 that still gives him double the space he needs. Why would you spend 200+ for space to just sit there sucking up electric?

Share this post


Link to post
Share on other sites

I am not exactly sure on the point of this thread either.. I tried to ask why he was so zero'd in on raid 5 for his small network setup when seems to me there are more cost effective and more flexible ways to access his storage in his other thread where he was looking for nas that supported raid 5.

He seems bent on spending money for no reason if you ask me - from the other thread it seems he has 1TB of data.. If he hell bent on parity/redundancy for his 1TB of data than a 2X2TB raid 1 would be better suited than a 4x3TB raid 5.. That is just a lot of space for no reason. Unless he is going to ramp up to close to 9TB of used space in say the next 6 months or so he is paying for spindles to spin for no reason.

I prob could of gotten by with adding only a 2TB disk, only reason I went with 3 was got a good price on it - and it only a few dollars more than the 2TB were going for. Got it for 109 when ST2000DM001 (2TB version is $99)

But even at that price he talking $440 for his raid 5 vs $200 for his raid 1 that still gives him double the space he needs. Why would you spend 200+ for space to just sit there sucking up electric?

 

I am all for redundancy on a file system to minimise downtime and the PITA it is to retrieve everything from backups.

 

However, proper RAID requires a proper controller and proper disks which you have to pay over the odds for for 9TB over a 4 disk RAID5 array would set you back circa ?400+ for a decent controller, and then ?120 per disk (cheap, WD Red 3TB) to ?225 per disk (better, WD RE 3TB), so between ?800 to ?1500, depending on where you want your budget.

 

Compare that to a consumer grade controller (?50 or so) and 4 consumer level disks at ?100 per disk (WD Green 3TB) with a software solution like DrivePool or even running NAS appliance such as freeNAS comes in at a whopping ?450.

 

Personally I would rather save the best part of ?400.

 

I doubt very much that anyone needs the performance level of enterprise grade RAID5 in their house (or even their small business to be frank)... but some people just seem to have money to burn - or completely a misguided view of what they really need.

Share this post


Link to post
Share on other sites

" but some people just seem to have money to burn - or completely a misguided view of what they really need."

Agree 100% on those points to be sure!!

And I agree it can be PITA to recover from backup.. But if done correctly you can run without any raid level and not have to recover from backup. My critical files are redundant.. They are on every disk in my pool.. If a disk failed, my critical files would be there online still. Only the files that were actual on the failed disk would have to be restored from backup.

And the thing is with the nature of media in a home setup, they might not even be restored - or they might take months to put back. They are not critical to me!! They are just nice to have my figure tips vs having to pull out the CD or the DVD or BluRay, etc. But I might never put that digital rip of scarface back online.. Or maybe I rerip my Europe 72 GD CD 3 months from now when in the mood to listen too it.

And since I am actively monitoring the health of the disks, and replace them as my space grows with newer larger disks - the old disk are quite likely to be removed from the storage before they fail.

The other point with his size of Raid - with hey he wants parity!! Fine, but from math -- especially on non enterprise disks.. If he lost 1 disk, its about a 50% shot that his raid will not rebuild anyway.. And now he has lost everything on that array and has to restore is critical stuff.. Why do this for non critical files that need are just nice to have online.

I would much rather spend the money on more media for my library, or more latest tech toy vs space that is going to sit there not being used ;)

But he did not seem to even want to spend a day or two researching other newer tech, which is another thing that blows my mind about some people.. Why not look - maybe it is better, maybe its not.. But its pretty cost friendly to research! What is worse thing that happens if you take a few weeks, what the cost of that tech you were going to buy is not going to go up!! As tech becomes yesterday it only gets cheaper! So that raid 5 card you were looking at might be 5 quid cheaper next week, etc.

Share this post


Link to post
Share on other sites

Hello,

Sure --- don't see how it would see the difference with a real disk a virtual one.  I am running in a vm currently, with the disks just passedthru to the vm.

Well, I might then make a VM and play around with it :)

 

 

Neither are other approaches such as DriveBender, DrivePool, Windows Storage Spaces, UnRAID etc.

I thought that this thread was about why 'standard' RAID levels were outdated and somewhat useless in the modern age.

This thread was guided toward standard RAID levels because most controllers and/or NAS boxes, only support standard RAID levels.

Example: Synology doesnt support (AFAIK) RAID-Z on its NAS boxes because it isnt standard. I wanted a apples-to-apples comparision where you dont have to for example buy/make your own FreeNAS custom box to support RAID-Z. Don't get me wrong: RAID-Z is great, its just not possible with a lot of hardware.

 

 

I am not exactly sure on the point of this thread either.. I tried to ask why he was so zero'd in on raid 5 for his small network setup when seems to me there are more cost effective and more flexible ways to access his storage in his other thread where he was looking for nas that supported raid 5.

The main point of the thread was to let me (and others) see better methods of parity storage without RAID5.

The issue is, BudMan, that you discard parity (for different logical reasons) and rely on a drive replacement and backups. I rather make backups and when a drive fails, simply replace it and not have to do anything else.

The risk does indeed come if one of those bits fail and the whole array is ruined. Im talking about my RAID5 non critical data.

He seems bent on spending money for no reason if you ask me - from the other thread it seems he has 1TB of data.. If he hell bent on parity/redundancy for his 1TB of data than a 2X2TB raid 1 would be better suited than a 4x3TB raid 5.. That is just a lot of space for no reason. Unless he is going to ramp up to close to 9TB of used space in say the next 6 months or so he is paying for spindles to spin for no reason.

Yes, currently Im around 1-1.5TB of data; Important, there is problably 10GB at max? The rest is either replacable or VMs.

 

 

I am all for redundancy on a file system to minimise downtime and the PITA it is to retrieve everything from backups.

Its all about PITA for me; I just rather plug in a drive, rebuild and be done. Not look thru the backup media and having to restore it to a new drive.

 

However, proper RAID requires a proper controller and proper disks which you have to pay over the odds for for 9TB over a 4 disk RAID5 array would set you back circa ?400+ for a decent controller, and then ?120 per disk (cheap, WD Red 3TB) to ?225 per disk (better, WD RE 3TB), so between ?800 to ?1500, depending on where you want your budget.

Its one of the reasons I wanted a NAS box; The RAID controller I have now is a complete POS. I dont trust it one bit (that came out as a unwatned joke). I believe (correct me if Im wrong) that the RAID controller on a NAS box is a lot better than some cheap RAID card I got. Also, my drives are not NAS oriented at all. Just your standard drives (dont remember the model right now)

 

I doubt very much that anyone needs the performance level of enterprise grade RAID5 in their house (or even their small business to be frank)... but some people just seem to have money to burn - or completely a misguided view of what they really need.

I had few intrest in the performance level of the RAID5. I just needed it to store my data decently. I read that RAID5 has OK performance (nothing like a SSD, obviously) but that really wasnt my choice behind choosing RAID5.

I feel that I have a completely misguided view of what I really need.

 

 

And I agree it can be PITA to recover from backup.. But if done correctly you can run without any raid level and not have to recover from backup. My critical files are redundant.. They are on every disk in my pool.. If a disk failed, my critical files would be there online still. Only the files that were actual on the failed disk would have to be restored from backup.

Ill be researching this disks in a pool method that you mention BudMan. It seems intresting but I need to know how it works and what exactly I need it to work.

The other point with his size of Raid - with hey he wants parity!! Fine, but from math -- especially on non enterprise disks.. If he lost 1 disk, its about a 50% shot that his raid will not rebuild anyway.. And now he has lost everything on that array and has to restore is critical stuff.. Why do this for non critical files that need are just nice to have online.

I guess I have been very lucky with that; A HDD has failed, been replaced and everything great. But with the write hole, I agree it is a huge risk (for the RAID)

But he did not seem to even want to spend a day or two researching other newer tech, which is another thing that blows my mind about some people.. Why not look - maybe it is better, maybe its not.. But its pretty cost friendly to research! What is worse thing that happens if you take a few weeks, what the cost of that tech you were going to buy is not going to go up!! As tech becomes yesterday it only gets cheaper! So that raid 5 card you were looking at might be 5 quid cheaper next week, etc.

It wasnt that I didnt want to spend a day or two seraching. First, this RAID5 has been active for 2 years more or less, I believe. Later, I had NO idea about these pools (still dont) so I only looked for RAID related information. I cant search for what I dont know!

Could you pinpoint some information on how to work with these pools? What do I need? This will help me look for information and mount it in a VM....

Thank you for all your help and information.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0

  • Recently Browsing   0 members

    No registered users viewing this page.