I need a portable way to move 40TB of data without using the cloud.


Recommended Posts

So what are you going with... If 40TB is estimate your going to need something BIGGER!!

 

And thanks for the info - has killed the curiosity cat.. But wouldn't you be able to just bring the "stuff" that the possible customer is interested in.. Not everything! ;)  Some smaller subset should be enough I would think.. Say your typical 10TB or so that could just be a single external low cost external disk.

 

And sure while your originals could be highest quality possible - when showing to a possible customer.. Does it really need to be 4K video for example.. Sure a nice png would prob do vs RAW ;)

Link to comment
Share on other sites

4 hours ago, BudMan said:

Some smaller subset should be enough I would think.. Say your typical 10TB or so that could just be a single external low cost external disk.

That's what I would do, but its not my decision. 

Link to comment
Share on other sites

Not sure how a customer is going to look through 40TB of data at any specific meeting or presentation..   I don't put my whole plex library on my phone when going on a flight... I sync a few of them for the flight ;)

 

I am just guessing but when you setup a presentation, I have to assume they are interested in specific sort of items you might sell, not your whole catalog of everything you have ever sold ;) heheh

 

Sure you might include a few key examples of different items or past things, etc. But the whole freaking library of everything going back to the 1930..

 

Tell you need that 60 or 100TB SSD.. Curious if they change there minds when they get the price tag for that ;)

Link to comment
Share on other sites

3 minutes ago, BudMan said:

Curious if they change there minds when they get the price tag for that ;)

I will let you know. I am meeting with them in about a hour for lunch. :) 

Link to comment
Share on other sites

I just wanted to say thanks to everyone for your help. Today I sat down and basically gave them 3 options and my opinion of what they should do and how to do it. None of them meet the 40TB they wanted, but the drobo comes close.

 

There is nothing funnier than watching sticker shock set in (especially when its family :D). Here they are with the stupid code names I made:

 

The Full Monty (about 36TB):

 

A Drobo 5D3.

WD Red 10TB NAS Hard Disk Drives  5 for the device and one extra for redundancy. 

 

Cost: $2,150 without tax

 

The Mad Max (2 drives with the same data).

 

Novus 14TB External USB-C (3.1,Gen2) Rugged Desktop Hard Drives

 

Cost: $1218 without tax

 

The Leo Laporte (1 set of 2 hard drives with a a copy of each drive).

 

WD Elements 10TB USB 3.0 Desktop Hard Drives

 

Cost: $780 without tax

 

 

I will let you know when a decision is made. I have a feeling it will be later today. 

 

 

Link to comment
Share on other sites

How does the full monty only work out to 36T??  5 x 10 should give you like 46..  You running it in raid with hot spare?  Seems wasteful for something that is used to just move data back and forth and not actual storage.. I would put them in raid 0.. This use case makes no sense for parity..  Or if you don't want 0, then jbod.. But seems pointless to waste space on parity.

 

Why did you not cost out the 50TB SSD? ;)

 

 

Link to comment
Share on other sites

6 minutes ago, BudMan said:

How does the full monty only work out to 36T??  5 x 10 should give you like 46..  You running it in raid with hot spare?  Seems wasteful for something that is used to just move data back and forth and not actual storage.. I would put them in raid 0.. This use case makes no sense for parity..  Or if you don't want 0, then jbod.. But seems pointless to waste space on parity.

 

Why did you not cost out the 50TB SSD? ;)

 

 

Because of the redundancy on the drobo. Here is their calculator: https://www.drobo.com/storage-products/capacity-calculator/

 

For the 50TB SSD. I wanted to see sticker shock not heart attacks. :)

Link to comment
Share on other sites

Yeah I can understand why you loose storage space with parity... My point is for this use case there is no point of parity.. Your just throwing money down the drain, and then having a hot spare as well.. Makes no sense for this use case..

Link to comment
Share on other sites

Don't forget a nice hard case to protect a Drobo... or a bunch of external hard drives and their power supplies!

 

And I'm not sure if you answered this already... but will this data only exist on these drives?  Or is it backed up somewhere else?  

 

Something about all your eggs in one basket... :p

 

 

EVDYg16.jpg

Link to comment
Share on other sites

Hello,

 

Note that I'm talking about using inexpensive shingled magnetic recording 2.5" 5.0GB HDDs, which can be found for about $100 now.  See https://www.enterprisestorageforum.com/storage-hardware/smr-drive-operation-2.html for an overview of this technology.  It might be better to think of using these more like tape drives than HDDs.

 

Doing backups to a 480GB NVMe SSD over a USB 3.1 gen 2 interface generally results transfers at a rate of 700-800 MB/s, which is much faster.  However, the amount of drives required for scaling this up would likely be out of @Xenon's budget.

 

Regards,

 

Aryeh Goretsky

 

21 hours ago, BudMan said:

3-4 TB a day??  That is horrible speeds.. USB3 is capable of way more than that... I can move files to my ds918+ nas at 220MBps (2x1gig internet using smb3 multichannel) -- so to move 40TB (if had that must storage) would take a bit over 2 days not weeks ;)

 

If your only seeing those speeds, you have something wrong... You should for sure be able to do 100MBps which would be more like 8.5GB per day.  But have to take into account this is prob 1,000s and 1,000s of files which yeah going to slow down transfer.

 

But yeah transfer of data even when local is a very good point.. Which is why I would look to reduction of the data size to kill both birds with 1 stone, more reasonable cost in transfer storage, and actual speed of transfer/access of data.

 

that 100TB SSD with 10ge connection would be ideal ;)

 

Link to comment
Share on other sites

Well the decision is made and the winner is............

 

The Drobo 5D3

 

But.....

 

It will only have 3 WD Red 10TB NAS Hard Disk Drives for a total of 18.5 TB. 

 

They decided to cut back on the data that will be on it, and if they need more space there is room for expansion. The expansion and ease of use was the deciding factor. As the drives get bigger we can just swap one out and put a new one in.

 

Total cost without tax $1397.

 

We have a 6TB hard drive in a computer that is not really being used so I will pull that out and it will be carried with it for emergencies.

 

As for the case, we use pelican cases a lot so I am gonna check them out to see if we can use one of them. If not.....

 

6 hours ago, Michael Scrip said:

Don't forget a nice hard case to protect a Drobo... or a bunch of external hard drives and their power supplies!

 

And I'm not sure if you answered this already... but will this data only exist on these drives?  Or is it backed up somewhere else?  

 

Something about all your eggs in one basket... :p

 

 

EVDYg16.jpg

What case is that? Do you have the manufactures name?

 

The data will be elsewhere too. :) I may be nuts, but i'm not crazy! 

Link to comment
Share on other sites

On 7/30/2019 at 10:40 PM, goretsky said:

Hello,

 

One thing to keep in mind is that it is going to take some time to transfer 40TB of files onto external storage, regardless of whether it is networked- (NAS) or directly-attached storage (DAS).  I believe I get about 3-4 TB/day when copying over a USB 3 interface to a two-drive RAID-0 enclosure with 5400 RPM SMR HDDs, so even using 7200 RPM PMR HDDs it's going to take at least a week or two, uninterrupted, to perform the copy.  The good news is that afterwards, syncing it each day/week/{whatever interval you choose} should be much quicker.

 

Regards,

 

Aryeh Goretsky

 

 

Yea I took that into consideration. The plan is to start adding data on a Friday morning and check it Monday morning. :)

 

I have a feeling that choosing what data they want on it will take longer than that. 

 

They want it operational for the beginning of September so I have time.

Link to comment
Share on other sites

2 hours ago, freqnasty said:

You’re a business that sells artwork, artefacts and antiquities?

Not as grandiose as you put it. :) 

 

We sell antiques. We specialize in high end glass items. Everything from vases, candle holders, perfume bottles, glass ware (plates bowls etc), stem ware (wine, champagne glasses etc) to stain glass windows. Part of what we do is find items that clients want (my Job). If someone breaks a glass in a 1914 stem ware set. We will identify the set and find a replacement glass. We also offer services for customers to document their collections for insurance purposes.

 

For the last couple of years we been acquiring books, maps, manuscripts, letters etc. We are hoping to enter that market in a few years. 

 

Its funny because I originally didn't go in to the business. I did IT for a financial company for over a decade. That was over 15 years ago. Man I feel old now.

  • Like 2
Link to comment
Share on other sites

3 hours ago, Xenon said:

What case is that? Do you have the manufactures name? 

I found that case by Googling "Pelican NAS" 

 

This particular case is no longer available... but it was a kit your could get with the Lacie 5big drive.  It is designed to carry the NAS with its drive inside.   I'm guessing this Lacie drive is built to handle the rigors of frequent travel.

 

Any deep Pelican case can carry a Drobo.  Just look for one with the proper internal dimensions and cut the foam to fit.

 

Some questions though... is a Drobo tough enough to carry around often?  Will you be transporting it daily?  Weekly?  It kinda freaks me out to think about carrying around a populated drive array.  :p

 

Something like this is specifically built to be transported:  LINK

Link to comment
Share on other sites

6 hours ago, Xenon said:

will only have 3 WD Red 10TB NAS Hard Disk Drives for a total of 18.5 TB. 

Your wasting money with parity in this scenario.. And once you do parity on the drives - not sure how you think your just going to add space or swap out a drive.. While you might be able to expand the array - going to take FOREVER!!!  if anything like the synologies...

 

Why not just put your 3 disks in JBOD?  So you would get the full amount of space vs wasting a whole disk on parity?  Or just 3 disks, so then sure you can take 1 disk out and put in bigger one, or add another disk. And then you just need to move the files around or add files to the thing vs having to wait for it to redo all the parity and striping if you add a disk to an array with parity.

 

BTW: lets us know how the transfer speeds to the thing are.. I have read that they are not anywhere as fast as they state they are..  And using any sort of stripped parity is going to be a hit on performance writing data to the thing.

Link to comment
Share on other sites

5 hours ago, Xenon said:

Not as grandiose as you put it. :) 

 

We sell antiques. We specialize in high end glass items. Everything from vases, candle holders, perfume bottles, glass ware (plates bowls etc), stem ware (wine, champagne glasses etc) to stain glass windows. Part of what we do is find items that clients want (my Job). If someone breaks a glass in a 1914 stem ware set. We will identify the set and find a replacement glass. We also offer services for customers to document their collections for insurance purposes.

 

For the last couple of years we been acquiring books, maps, manuscripts, letters etc. We are hoping to enter that market in a few years. 

 

Its funny because I originally didn't go in to the business. I did IT for a financial company for over a decade. That was over 15 years ago. Man I feel old now.

Don't have anything to add to the topic, but that sounds super interesting! 

 

Always nice to see those on the site that don't do that bog standard IT (not that there's anything wrong with that 😛)

Link to comment
Share on other sites

But keep in mind that with three 10 TB disks you can make a RAID0 array of 30 TB. Or even better: do not make a RAID system at all, but tread them as Just a Bunch of Disks. Why? Your data does not need the redundancy on this system. You have all your data at home, backed up and all. If you are on the road (at the customer) and you notice that one disk is dead, you won't be able to rebuild the RAID system because you need a fourth disk and it takes to long. With JBOD you will have two working disks so the chance that the data you need is available will actually increase by not having a RAID. You only have a 33% chance that the data you need is on the broken disk. (if all disks are filled equally). There is however a better sollution. Do not buy a Drobo, but buy three external hard disks. Why. Because if the Drobo itself fails, all three disks will not be available. With three external hard disks you will always have two disks if the case fails. You could even open a case and swap the disk with the faulty case, so there could still have alle data available.

Link to comment
Share on other sites

7 hours ago, Michael Scrip said:

I found that case by Googling "Pelican NAS" 

 

This particular case is no longer available... but it was a kit your could get with the Lacie 5big drive.  It is designed to carry the NAS with its drive inside.   I'm guessing this Lacie drive is built to handle the rigors of frequent travel.

 

Any deep Pelican case can carry a Drobo.  Just look for one with the proper internal dimensions and cut the foam to fit.

 

Some questions though... is a Drobo tough enough to carry around often?  Will you be transporting it daily?  Weekly?  It kinda freaks me out to think about carrying around a populated drive array.  :p

 

Something like this is specifically built to be transported:  LINK

Thanks for the info! 

Link to comment
Share on other sites

This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.