Is putting everything in one huge 25GB file a bad idea?


Recommended Posts

55 minutes ago, Danielx64 said:

The latest Deus Ex has like one massive archive file and several smaller ones. I cant decide if it a bad idea or a good one. I guess that with that huge file you would everything in one block on the harddrive so that performance would be good but IDK.

It depends.  As I understand it, it's good for the applications internal memory management.

 

Lots of first person games use large files such, though perhaps not that large.

45 minutes ago, LostCat said:

It depends.  As I understand it, it's good for the applications internal memory management.

 

Lots of first person games use large files such, though perhaps not that large.

Looks like I need to find a way of getting that large file in one block on the harddrive. Mind you, The Division is quite large but use lots of small files.....

Seems like it would help performance generally and I guess it would help with reducing the game's disk space footprint? Honestly though, being one that likes to poke around in game assets and tweak stuff, I rather preferred when games had looser structures to their data. Unreal engine titles prior to version 3 were great like this, and Quake 3/Doom 3 pk3/4 files were just zips that you could open and modify at will. That was awesome.

 

While I'm sure there's some performance upsides to keeping game data in one big file, I'd imagine the big driver towards the one-big-impenetrable-file structure is to prevent tampering, especially in this age of Denuvo and built-in DLC.

 

I did notice that Mankind Divided takes a hell of a long time to load, but the maps are huge and transitioning around them seems totally seamless.

1 minute ago, Circaflex said:

What exactly do you mean?

Like defragging the entire harddrive but just that one file :)

2 minutes ago, IanHead said:

I did notice that Mankind Divided takes a hell of a long time to load, but the maps are huge and transitioning around them seems totally seamless.

I'm having serious issues with one area on the map where I would only get like 1 frame per second :( Never done that till now.

2 hours ago, Danielx64 said:

Looks like I need to find a way of getting that large file in one block on the harddrive. Mind you, The Division is quite large but use lots of small files.....

I'm just sayin you can go all the way back to FEAR 1 and perhaps before that to find insanely large files in FPS land.

13 minutes ago, LostCat said:

I'm just sayin you can go all the way back to FEAR 1 and perhaps before that to find insanely large files in FPS land.

Yeah, FEAR has those .Arch## files, though iirc it is split up into smaller chunks rather than one enormous one.

1 hour ago, Danielx64 said:

Like defragging the entire harddrive but just that one file :)

use Mark Russinovich's Contig or use Piriform Defraggler which also have similar capability (via its search feature).

Edited by Torolol

So you have bad performance in one particular area of a map in a particular game... You think it has to do with the area that file is located? Then wouldn't that imply hardware, not some file system anomaly? Just trying to follow the logic here... Instead of defragging, perhaps  a checkdisk is in order. Or it's just a bug.

On 2016-08-29 at 8:07 PM, Danielx64 said:

The latest Deus Ex has like one massive archive file and several smaller ones. I cant decide if it a bad idea or a good one. I guess that with that huge file you would everything in one block on the harddrive so that performance would be good but IDK.

For backup it's not cool. Every time the game is updated you have to redo the whole 25GB backup. If the backup is done online or using an external disk on an old computer supporting USB2 only that's not cool.

 

Also if the file gets corrupted you have to redownload the whole thing (if you don't have a backup).

15 minutes ago, LaP said:

For backup it's not cool. Every time the game is updated you have to redo the whole 25GB backup. If the backup is done online or using an external disk on an old computer supporting USB2 only that's not cool.

 

Also if the file gets corrupted you have to redownload the whole thing (if you don't have a backup).

If you're still using USB2 or flaky drives but playing high end games like Deus Ex: Mankind Divided...that is weird.

3 hours ago, LostCat said:

If you're still using USB2 or flaky drives but playing high end games like Deus Ex: Mankind Divided...that is weird.

Why?

 

My PC can still run new games. I wont upgrade it just to get USB3 support ;)

 

Not my fault if cpus did not progress much in the last 10 years. My overclocked core i5 750 is still doing the job at 1080p while paired with a 970. It is starting to show its age lately but it's still good enough.

It's not the packaging but the usage that's important. Guild Wars 2 is one 27GB asset file but has no loading issues. On the other hand, SWTOR takes minutes just to load the tiny interior of your personal starship.

  • Like 2
This topic is now closed to further replies.
  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Posts

    • Qualcomm takes on NVIDIA with new Dragonfly CPU and AI chips by Pradeep Viswanathan Microsoft, Google, Amazon, AMD, Meta, Apple, OpenAI, and several others have been developing their own chips for AI infrastructure. However, NVIDIA still remains the dominant player in the market. Today, Qualcomm announced a major expansion of its data center infrastructure portfolio to better compete with NVIDIA. The new lineup includes the Qualcomm Dragonfly C1000 CPU, Qualcomm High Bandwidth Compute technology, the Dragonfly AI300 inference accelerator, new connectivity products, and custom silicon solutions. Qualcomm claims that this new lineup improves performance per watt, token throughput, and total cost of ownership for AI data centers. The Dragonfly C1000 is a new data center CPU built with Qualcomm’s custom Oryon cores. This chip will feature more than 250 cores, frequencies above 5GHz, and a chiplet-based design. Qualcomm claims that this new C1000 can deliver more than 2x better performance per watt compared to existing server CPU offerings based on specifications. The Dragonfly C1000 will support PCIe Gen 7 with more than 2TB/s of connectivity, along with CXL, advanced RAS features, and both air and liquid cooling. Qualcomm expects the Dragonfly C1000 to be commercially available in 2028. Additionally, Qualcomm and Meta announced a multi-year, multi-generation agreement under which Qualcomm will supply Dragonfly C1000 data center CPUs for Meta’s next-generation server fleet. Qualcomm also announced High Bandwidth Compute, a new near-memory computing architecture designed to address AI’s memory bandwidth bottleneck. HBC Gen 1 will debut with the Dragonfly AI250, which is expected to sample in mid-2027. The AI250 will deliver 133TB/s per card, an 18x increase in effective memory bandwidth compared to the AI200 with LPDDR5X. The new Dragonfly AI300 with HBC Gen 2 is a rack-level AI inference platform from Qualcomm. Qualcomm claims that the AI300 can deliver 4x to 8x better performance per watt compared to existing GPU-based architectures based on memory bandwidth per watt per card. The Dragonfly AI300 is expected to be available in 2028.
    • IBM reveals sub-1nm chip technology, production expected in another 5 years by Pradeep Viswanathan TSMC is now leading the chip manufacturing industry with its 2nm-class process node called N2. Samsung Foundry also has a 2nm-class process node called SF2. TSMC says N2 entered volume production in Q4 2025. Samsung says SF2 started mass production in 2025. Today, IBM announced the world’s first sub-1-nanometer chip technology, marking another major semiconductor research milestone. The new technology is based on a 0.7nm, or 7-angstrom, node and uses a new transistor architecture called “nanostack.” The new design vertically stacks and staggers nanosheet-based transistors so that more components can fit into the same chip area while also improving performance and power efficiency. IBM claims that this new sub-1nm chip can pack nearly 100 billion transistors onto a chip the size of a fingernail. This offers almost twice the density, up to 50 percent higher performance, or 70 percent better energy efficiency when compared to IBM's 2nm node design announced back in 2021. Also, IBM mentioned that this new architecture can deliver 40 percent SRAM scaling. It is important to consider that this announcement from IBM is a research milestone rather than a near-term process node launch. Back in 2021, IBM unveiled the world’s first 2nm chip design, claiming 50 billion transistors on a fingernail-sized chip and major performance and efficiency gains. Five years later, IBM’s 2nm technology has still not entered mainstream commercial production. That is because IBM is no longer a major commercial chip manufacturer. It sold its chip manufacturing business to GlobalFoundries years ago and has since then focused only on semiconductor research, IP development, and partnerships. To productize its 2-nm chip technology, IBM partnered with Japan’s Rapidus, but it has not resulted in anything shipping at scale. IBM says that its new sub-1nm technology can reach production as early as within the next five years. If that happens, it will likely depend on manufacturing partners, advanced EUV tooling, and years of yield improvements.
  • Recent Achievements

    • Week One Done
      Meta Plast earned a badge
      Week One Done
    • First Post
      kinowa earned a badge
      First Post
    • Rookie
      krychek57 went up a rank
      Rookie
    • Grand Master
      Jaybonaut went up a rank
      Grand Master
    • One Year In
      Philsl earned a badge
      One Year In
  • Popular Contributors

    1. 1
      +primortal
      455
    2. 2
      +Edouard
      170
    3. 3
      PsYcHoKiLLa
      135
    4. 4
      Michael Scrip
      78
    5. 5
      Xenon
      77
  • Tell a friend

    Love Neowin? Tell a friend!