I need a portable way to move 40TB of data without using the cloud.

By Xenon
in Hardware Hangout

 Share

Recommended Posts

Mentor

Sylar0

Posted
Posted
4 hours ago, Xenon said:

To be as vague as possible. Photos of items we sell/sold (dating back to the early 1930's) and the clients past purchases. The clients info will only be on the device until it is not needed and then securely deleted. Of course the whole thing will be encrypted. 

Ha ha I know when to take a hint. You know how to make a man even more curious however.

Link to comment
Share on other sites
Grand Master

+BudMan MVC

Posted
  • MVC
Posted

So this is work related, for a company that has been selling ###### since the 1930's etc.. and the items are so "sensitive" that you can not just store the image online... An you only have a budget of $1500?  So did you drop a zero there?  you mean you have a budget of 15K?  Maybe 2 zeros?

 

What idiot set this budget?

 

I thought this was some ma and pa shop, or your own personal stuff that needed to move some data on the cheap?

Link to comment
Share on other sites
Grand Master

goretsky Supervisor

Posted
  • Supervisor
Posted

Hello,

 

I have not used a NAS with 40TB of capacity, myself, but here are some options to get you started:

 

  • Drobo 5N2 NAS (5×3.5" bays) [starting at $500]
  • QNAP TS-453Be (4×3.5" bays) [starting at about $450]
  • SuperMicro SuperServer 5029C-T (4×3.5", 2×2.5" bays) [starting at about $450]
     

One thing that I will mention right away is that you are likely going to be exceeding your budget of $1,500.  Looking around at the cost of 10TB SATA HDDs, they seem to go for $250-330 each, depending upon model and features, and 12TB SATA HDDs go for $330-500 each, so you might want to add another few hundred dollars to your budget, especially if you need to add RAM, an additional NIC, or spare drives in case one of the ones installed in the NAS fails, or you want a semi-custom storage case for transporting the NAS between locations.

 

Regards,

 

Aryeh Goretsky

Link to comment
Share on other sites
Grand Master

Michael Scrip

Posted
Posted
On 7/26/2019 at 5:57 PM, Xenon said:

Yea. I'm actually starting to lean toward 5 8TB drives. For organization it would be easier than 4 10TBs. Then spend a lot on a travel case.

Since you used the word "move" in the thread topic... I'm guessing you're not leaving a copy wherever all this data is stored now.  Will this data only exist on these portable drives?

 

You also mention reliability.  Is this data backed up at all right now?

 

I ask because if you think you need 5 8TB drives... you should probably get 10 8TB drive just for redundancy/backup.

 

You'd hate to drop your travel case and some of the drive go bye-bye.

Link to comment
Share on other sites
Veteran

DentedAphid7

Posted
Posted
On 7/27/2019 at 4:52 AM, Circaflex said:

What type of connection are you looking to have?

 

He specifically states he doesnt want anything cloud based, i would assume he doesnt want this to be on the internet.

Have you herd of intranet?

 

On 7/27/2019 at 7:49 AM, Xenon said:

To be as vague as possible. Photos of items we sell/sold (dating back to the early 1930's) and the clients past purchases. The clients info will only be on the device until it is not needed and then securely deleted. Of course the whole thing will be encrypted. 

Why not get one of those HDD docks on AliExpress. They are far cheaper (I have the Orico one) and reliable so far.

Link to comment
Share on other sites
Grand Master

Circaflex

Posted
Posted
1 minute ago, DentedAphid7 said:

Have you herd of intranet?

 

Why not get one of those HDD docks on AliExpress. They are far cheaper (I have the Orico one) and reliable so far.

Of course I have, however if you happened to read the reply that I was responding to, you would see the mentioned "net." Their follow up was also referring to internet via RDP from two different sites.

  • DentedAphid7
  • Like 1
Link to comment
Share on other sites
Grand Master

Xenon

Posted
  • Author
Posted
9 hours ago, DentedAphid7 said:

Have you herd of intranet?

 

Why not get one of those HDD docks on AliExpress. They are far cheaper (I have the Orico one) and reliable so far.

😀 You know I just researching this: 

 

http://www.orico.cc/us/product/detail/253.html

  • DentedAphid7
  • Like 1
Link to comment
Share on other sites
Neowinian

Ectatomma

Posted
Posted (edited)

You said you don't want the info to be on the cloud, so I could assume it's because you're concerned about the privacy of your data. Have you heard about Nextcloud? You can setup your own server, so no big corporations/governments spying on your data. Connection to the server could be through a VPN running on the same server, so it's very unlikely someone eavesdropping. It support encryption as well. At least it could be useful as a backup solution (What if you loose/drop the storage device?). It would be possible to setup using consumer-level hardware, so it'll be near your budget (there are NAS solutions that support Nextcloud as well).

It's a must for you to carry all the data all the time?

Edited by Ectatomma
  • Mindovermaster
  • Like 1
Link to comment
Share on other sites
Neowinian

Ectatomma

Posted
Posted (edited)
1 hour ago, Xenon said:

😀 You know I just researching this: 

 

http://www.orico.cc/us/product/detail/253.html

It's important to consider redundancy of your hard drive array. If one of your drives fails, what's going to happen? So, it's not same to have 5 10TB separate hard drives as 5 different storage units (looks like the enclosure you point on the link is that way) or to have all these drives grouped together as a single storage device using what it's known as RAID. RAID allows to have one drive (RAID 5) or two of drives as a redundant drives (RAID 6), so if one (or up to two drives on RAID 6) drives fail, you don't loose your data. Most of the NAS and some enclosures have this feature. Redundancy means storage capacity (and writing  performance) is reduced, so on a 5  10TB drive RAID 5 array you will have roughly 40 TB available and 30 TB on a RAID 6 (the data is distributed and replicated on all the drives in order to be redundant). Once a drive fails and you replace it, the array should be rebuilt in order to regain redundancy (that takes a lot of time), so on a RAID 5 there is a chance for another drive to fail while rebuilding the array and loose all your data (that tend to happens whit drives of the same lot), there is where RAID 6 is less prone to catastrophic data loss, as if another drive fails while it's rebuilding, data is still accessible. There is also RAID 0: all drives grouped as one, high performance, no redundancy, one drive fail, you loose all data; JBOD: drives grouped as a single device, one after the other, so if one drive fails, you only loose data only on that drive.

Some NAS are linux based (i. e. QNAP) and them use a different system for redundancy called ZFS, which achieve the same as RAID 5 or 6 according to configuration (single or double parity respectively) and it's less prone to data corruption.

So at the end, it would be better to build your your NAS using 6 10TB drives on RAID 6 or ZFS double parity in order achieve 40 TB and to protect your data from drive failure, and try to buy drives coming from different lots (different sellers?). The same could apply to a backup solution (you must have a backup and a recovery plan)

Edited by Ectatomma
Link to comment
Share on other sites
Grand Master

Xenon

Posted
  • Author
Posted
12 hours ago, goretsky said:

Hello,

 

I have not used a NAS with 40TB of capacity, myself, but here are some options to get you started:

 

  • Drobo 5N2 NAS (5×3.5" bays) [starting at $500]
  • QNAP TS-453Be (4×3.5" bays) [starting at about $450]
  • SuperMicro SuperServer 5029C-T (4×3.5", 2×2.5" bays) [starting at about $450]
     

One thing that I will mention right away is that you are likely going to be exceeding your budget of $1,500.  Looking around at the cost of 10TB SATA HDDs, they seem to go for $250-330 each, depending upon model and features, and 12TB SATA HDDs go for $330-500 each, so you might want to add another few hundred dollars to your budget, especially if you need to add RAM, an additional NIC, or spare drives in case one of the ones installed in the NAS fails, or you want a semi-custom storage case for transporting the NAS between locations.

 

Regards,

 

Aryeh Goretsky

Thanks for the recommendations. I have had more than a few people recommend dobro. The 5D3 would probably be better for us. 

 

As for the budget, I just had a long hard talk with the person who controls the budget and I now have some leeway with it. 

 

Link to comment
Share on other sites
Grand Master

xrobwx71

Posted
Posted
On 7/26/2019 at 3:58 PM, Steven P. said:

Hmm have you looked at any of Synology's offerings? https://www.neowin.net/news/tags/synology/ we've reviewed quite a lot of them and they are rugged, within your budget, does what you are looking for.

 

Example https://www.neowin.net/news/review-of-the-synology-ds1019-a-storage-workhorse-and-a-plex-beast

(I'm speaking for Synology in general not suggesting the DS718 for 40TB.) We are using the DS 718+ for offsite storage.  It's been flawless. I don't see the problem with traveling with it either. They are very sturdy and well made. The drives are locked in and well protected.  As long as the drives aren't spinning, they can stand up to a surprising amount of abuse.

  • Steven P. and Brandon H
  • Like 2
Link to comment
Share on other sites
Grand Master

Jason S. Global Moderator

Posted
  • Global Moderator
Posted
On 7/26/2019 at 5:57 PM, Xenon said:

Yea. I'm actually starting to lean toward 5 8TB drives. For organization it would be easier than 4 10TBs. Then spend a lot on a travel case.

 

And here's a question that should get a few responses. Seagate or WD hard drives. I used to always use hitachi's back in the day. 

you need to keep in mind formatting and RAID. You'll need to buy something like (5) 10TB drives to get 40TB usable space. After factoring the cost of a NAS plus (5) drives, you'll be well over your $1500 budget.

 

https://www.synology.com/en-us/support/RAID_calculator

 

btw - there have to be companies out there that handle services like this. i'm picturing a company that would come to the site and transfer the data to their solution, take it to the new location and offload it. Something like snowball (already mentioned) but local and not tied to the cloud.

Link to comment
Share on other sites
Grand Master

sc302 Veteran

Posted
  • Veteran
Posted

nothing is really going to be a true portable solution.  That is a lot of data to be carrying around at any given time.  

There are plenty 40TB NAS boxes, that would probably be the most portable.  

 

Link to comment
Share on other sites
Grand Master

+BudMan MVC

Posted
  • MVC
Posted

If you want "portable" go with SSD.. Just going to cost you a lot more ;)  A lot!!!  But they have 60TB SSD from seagate.. Not something you pick up off amazon.  But they can be had.. Viking has the 50TB drive..  But the cost is going to be a tad higher than your $1500, or prob even double that..

https://blog.seagate.com/intelligent/seagate-60tb-ssd-named-best-of-show-at-flash-memory-summit/

https://www.vikingtechnology.com/uhc-silo-campaign092017/

Link to comment
Share on other sites
Grand Master

Mindovermaster Moderator

Posted
  • Moderator
Posted

I've been reading this thread, but I just have one question... Do you NEED to carry around 40TB of data? Surely you only will be able to work on 1/8 of that... If that... Unless you have 100 monkeys working at it...

Link to comment
Share on other sites
Grand Master

+BudMan MVC

Posted
  • MVC
Posted

Also curious to the details of why you would need to move around that much data.. There was mention of raw files and "Photos of items we sell/sold (dating back to the early 1930's)"

 

If they are just images of your stuff you sell, why does the image have to be so large?  Could you not shrink them to a manageable size.. How many images are we talking?  1 mil, 10,000 million, 100 million?  You could store a freaking shitton of on a normal sized very reasonable sized portable drive..  Lets say images were 500K each.. At even 1TB your talking what 2k images..  A 10TB external is what like 200$ that would be 20K images if they were 500K each.. Those are some pretty big jpg or png files to be honest..

Link to comment
Share on other sites
Grand Master

goretsky Supervisor

Posted
  • Supervisor
Posted

Hello,

 

One thing to keep in mind is that it is going to take some time to transfer 40TB of files onto external storage, regardless of whether it is networked- (NAS) or directly-attached storage (DAS).  I believe I get about 3-4 TB/day when copying over a USB 3 interface to a two-drive RAID-0 enclosure with 5400 RPM SMR HDDs, so even using 7200 RPM PMR HDDs it's going to take at least a week or two, uninterrupted, to perform the copy.  The good news is that afterwards, syncing it each day/week/{whatever interval you choose} should be much quicker.

 

Regards,

 

Aryeh Goretsky

 

Link to comment
Share on other sites
Grand Master

+BudMan MVC

Posted
  • MVC
Posted

3-4 TB a day??  That is horrible speeds.. USB3 is capable of way more than that... I can move files to my ds918+ nas at 220MBps (2x1gig internet using smb3 multichannel) -- so to move 40TB (if had that must storage) would take a bit over 2 days not weeks ;)

 

If your only seeing those speeds, you have something wrong... You should for sure be able to do 100MBps which would be more like 8.5GB per day.  But have to take into account this is prob 1,000s and 1,000s of files which yeah going to slow down transfer.

 

But yeah transfer of data even when local is a very good point.. Which is why I would look to reduction of the data size to kill both birds with 1 stone, more reasonable cost in transfer storage, and actual speed of transfer/access of data.

 

that 100TB SSD with 10ge connection would be ideal ;)

Link to comment
Share on other sites
Grand Master

Mindovermaster Moderator

Posted
  • Moderator
Posted
19 minutes ago, BudMan said:

3-4 TB a day??  That is horrible speeds.. USB3 is capable of way more than nat... I can move files to my ds918+ nas at 220MBps (2x1gig internet using smb3 multichannel) -- so to move 40TB (if had that must storage) would take a bit over 2 days not weeks ;)

Caught you again. Think you meant "that"

 

(sorry, was to good to pass up...)

Link to comment
Share on other sites
Grand Master

Jim K Global Moderator

Posted
  • Global Moderator
Posted
6 minutes ago, Mindovermaster said:

Caught you again. Think you meant "that"

 

(sorry, was to good to pass up...)

*too

Link to comment
Share on other sites
Grand Master

Xenon

Posted
  • Author
Posted (edited)
13 hours ago, BudMan said:

Also curious to the details of why you would need to move around that much data.. There was mention of raw files and "Photos of items we sell/sold (dating back to the early 1930's)"

 

If they are just images of your stuff you sell, why does the image have to be so large?  Could you not shrink them to a manageable size.. How many images are we talking?  1 mil, 10,000 million, 100 million?  You could store a freaking shitton of on a normal sized very reasonable sized portable drive..  Lets say images were 500K each.. At even 1TB your talking what 2k images..  A 10TB external is what like 200$ that would be 20K images if they were 500K each.. Those are some pretty big jpg or png files to be honest..

I actually didn't mean to be mysterious. Sorry.  The powers that be want a single portable archive of everything we sell/sold and reference materials. The non video reference materials are currently about 8 TB (mostly old drawings, photos, books and a huge amount of old auction catalogs). There are close to 9TB of reference videos. As time went by the company took more reference photos for items and videos. And as digital photography got better, files sizes got bigger. We recently sold a 18th century french vase. For it alone there were over 50 16GB raw photos and a 4k video. The problem we have is that we must we must document the item completely and keep the data for insurance purposes. The 40TB was a estimate. The material we have now and as old photos are still being digitized are added. We are using a service for that. Also there is space for the future. 

 

The reason it must be portable and not have the information on the web is that this will be used mostly at clients locations to show them items and reference materials. We cant depend on a unknown internet connection or if they even have one. 

 

The original budget they gave was a joke. I was hoping someone here would know of something cheap and reliable.

 

I have pretty much chosen the hardware. Its just now just getting the budget for it. 

 

Link to comment
Share on other sites
This topic is now closed to further replies.
 Share
Go to topic listing

  • Recently Browsing   0 members

    • No registered users viewing this page.