Is putting everything in one huge 25GB file a bad idea?

[Danielx64]

The latest Deus Ex has like one massive archive file and several smaller ones. I cant decide if it a bad idea or a good one. I guess that with that huge file you would everything in one block on the harddrive so that performance would be good but IDK.

[spaceelf]

55 minutes ago, Danielx64 said:

The latest Deus Ex has like one massive archive file and several smaller ones. I cant decide if it a bad idea or a good one. I guess that with that huge file you would everything in one block on the harddrive so that performance would be good but IDK.

It depends.  As I understand it, it's good for the applications internal memory management.

 

Lots of first person games use large files such, though perhaps not that large.

[Danielx64]

45 minutes ago, LostCat said:

It depends.  As I understand it, it's good for the applications internal memory management.

 

Lots of first person games use large files such, though perhaps not that large.

Looks like I need to find a way of getting that large file in one block on the harddrive. Mind you, The Division is quite large but use lots of small files.....

[Torolol]

if its TAR/ISO file type, read-only with no rewriting into the big file and no-decompressing along the way? thats usually ok performance wise.

 

[IanHead]

Seems like it would help performance generally and I guess it would help with reducing the game's disk space footprint? Honestly though, being one that likes to poke around in game assets and tweak stuff, I rather preferred when games had looser structures to their data. Unreal engine titles prior to version 3 were great like this, and Quake 3/Doom 3 pk3/4 files were just zips that you could open and modify at will. That was awesome.

 

While I'm sure there's some performance upsides to keeping game data in one big file, I'd imagine the big driver towards the one-big-impenetrable-file structure is to prevent tampering, especially in this age of Denuvo and built-in DLC.

 

I did notice that Mankind Divided takes a hell of a long time to load, but the maps are huge and transitioning around them seems totally seamless.

[Circaflex]

1 hour ago, Danielx64 said:

Looks like I need to find a way of getting that large file in one block on the harddrive.

What exactly do you mean?

[Danielx64]

1 minute ago, Circaflex said:

What exactly do you mean?

Like defragging the entire harddrive but just that one file

2 minutes ago, IanHead said:

I did notice that Mankind Divided takes a hell of a long time to load, but the maps are huge and transitioning around them seems totally seamless.

I'm having serious issues with one area on the map where I would only get like 1 frame per second  Never done that till now.

[spaceelf]

2 hours ago, Danielx64 said:

Looks like I need to find a way of getting that large file in one block on the harddrive. Mind you, The Division is quite large but use lots of small files.....

I'm just sayin you can go all the way back to FEAR 1 and perhaps before that to find insanely large files in FPS land.

[IanHead]

13 minutes ago, LostCat said:

I'm just sayin you can go all the way back to FEAR 1 and perhaps before that to find insanely large files in FPS land.

Yeah, FEAR has those .Arch## files, though iirc it is split up into smaller chunks rather than one enormous one.

[Torolol]

1 hour ago, Danielx64 said:

Like defragging the entire harddrive but just that one file

use Mark Russinovich's Contig or use Piriform Defraggler which also have similar capability (via its search feature).

Edited August 30, 2016 by Torolol

[ANON86364]

So you have bad performance in one particular area of a map in a particular game... You think it has to do with the area that file is located? Then wouldn't that imply hardware, not some file system anomaly? Just trying to follow the logic here... Instead of defragging, perhaps  a checkdisk is in order. Or it's just a bug.

[spaceelf]

2 hours ago, IanHead said:

Yeah, FEAR has those .Arch## files, though iirc it is split up into smaller chunks rather than one enormous one.

There's more than one here as well...one of them is just very large.

[LaP]

On 2016-08-29 at 8:07 PM, Danielx64 said:

The latest Deus Ex has like one massive archive file and several smaller ones. I cant decide if it a bad idea or a good one. I guess that with that huge file you would everything in one block on the harddrive so that performance would be good but IDK.

For backup it's not cool. Every time the game is updated you have to redo the whole 25GB backup. If the backup is done online or using an external disk on an old computer supporting USB2 only that's not cool.

 

Also if the file gets corrupted you have to redownload the whole thing (if you don't have a backup).

[spaceelf]

15 minutes ago, LaP said:

For backup it's not cool. Every time the game is updated you have to redo the whole 25GB backup. If the backup is done online or using an external disk on an old computer supporting USB2 only that's not cool.

 

Also if the file gets corrupted you have to redownload the whole thing (if you don't have a backup).

If you're still using USB2 or flaky drives but playing high end games like Deus Ex: Mankind Divided...that is weird.

[LaP]

3 hours ago, LostCat said:

If you're still using USB2 or flaky drives but playing high end games like Deus Ex: Mankind Divided...that is weird.

Why?

 

My PC can still run new games. I wont upgrade it just to get USB3 support

 

Not my fault if cpus did not progress much in the last 10 years. My overclocked core i5 750 is still doing the job at 1080p while paired with a 970. It is starting to show its age lately but it's still good enough.

[spaceelf]

I assume you know you can get a cheap PCI express USB3 card.

 

Sure, throwing out the whole box just for that would be stupid, but putting a card in there is pretty bloody easy.

[Eric Veteran]

It's not the packaging but the usage that's important. Guild Wars 2 is one 27GB asset file but has no loading issues. On the other hand, SWTOR takes minutes just to load the tiny interior of your personal starship.