Windows 8: New "Protogon" filesystem could be the next big thing

            

We wish we knew exactly what's in store here, but we've found some exciting tidbits in Windows 8 that indicate that Microsoft's top-secret new filesystem -- Protogon -- is finally coming to fruition as part of Windows 8. 

If you've been following closely, we saw Protogon first come about back in June when a screenshot surfaced showing a drive with the file system "protogon" as the type.  Mary Jo Foley of ZDNet also reported about the technology in May, and found that Protogon looked like "a kernel mode driver for some sort of (yet unknown) file system called Protogon. It's unclear, whether this is a major new file system or just some minor subsystem."

On top of this, Rafael Rivera of Within Windows discovered that the technology "seems to incorporate database-like concepts like transactions, cursors, rows and tables" which sounds oddly familiar to the failed WinFS project that was meant to ship with Vista. Neowin decided to dig deeper in Windows 8 and uncovered evidence that the technology is there, but locked down right now.
 
In the developer preview of Windows 8, references to Protogon exist in the registry, especially in reference to "boot file system" as well as folders containing libraries for Protogon. We even managed to get the "format" utility in Windows 8 to attempt to format the drive in the Protogon format, and it recognized the file system type. A blog in Russian managed to format a drive into the Protogon format, and said that after some analysis, they were able to conclude that the format has some similarities to NTFS right now, but does not operate in the same way at all.

 

Steve Sinofsky said today that most of the new stuff in Windows 8 is disabled right now. Here's hoping someone will figure out how to unlock the hidden features. 

Report a problem with article
Previous Story

Razer to launch Xbox 360 arcade stick; seeks feedback

Next Story

Windows 8: Want the classic menu? There's an app for that

36 Comments

Commenting is disabled on this article.

Unfortunately for us the users, there is no true sucessor for the Fat32 filesystem, I mean... just look at all devices that have it, incorporating NTFS, I assume, must be far costly and closed.

Arceles said,
Unfortunately for us the users, there is no true sucessor for the Fat32 filesystem, I mean... just look at all devices that have it, incorporating NTFS, I assume, must be far costly and closed.

It's probably due to the complexity and speed as well, no? With embedded metadata there isn't a huge benefit trying to implement NTFS on a consumption device.

Arceles said,
Unfortunately for us the users, there is no true sucessor for the Fat32 filesystem, I mean... just look at all devices that have it, incorporating NTFS, I assume, must be far costly and closed.

FAT32-Ext?

NTFS is not just about the data structures, as it requires a bit of overhead to keep the structures consistent as well. Reading NTFS is easy, writing data 'properly' is where it gets far more complex than people realize, and why there is nothing that compares to it for functionality and speed. Moving around even 'saved' bit of a file seem harmless, but NTFS's states and journaling and transactions and encryption and compression make all of this hard.

Even FS drivers on Linux that support NTFS, can butcher NTFS volumes when they are put back into use in Windows, which Windows then has to sort out and repair in the background.

I look for Microsoft to have a duality concept up their sleeve that may be more than this test FS seen in Windows 8.

thenetavenger said,
FAT32-Ext?

exFAT.

exFAT is "FAT64" with some other structural issues fixed (like a free space bitmap that means the filesystem can actually calculate its free space in a timely fashion!)

Native since Windows Vista SP1. Available for XP via Hotfix. In MacOS since 10.6.8

There's transactional and ACL support on it too, but ACL part is unimplemented and the transactional part is only used on WinCE.

Azure makes use of some of these database filesystem concepts, too, because as far as I know you don't have access to the file system and have to store everything in tables. But part of Azure is not relational.

Running Windows in a VirtualBox in Fedora is like running a V8 engine inside a V4 engine.

Still, the pic says a lot. I'd be very interested in this new file system of Microsoft's. Now if they could just change that ugly Used Space graphic that has existed since XP...

And the change to using MS SQL (basically, SQL Server) also makes all sorts of sense, as SQL is not a flatfile database. Further, Microsoft *as a company* is moving away from non-SQL databases (if I'm not mistaken, Access 2010 will be the last non-SQL version of Access). NTFS and Index Server are pretty much the last remaining ties between Microsoft and flat-file databases.

PGHammer said,
Further, Microsoft *as a company* is moving away from non-SQL databases (if I'm not mistaken, Access 2010 will be the last non-SQL version of Access). NTFS and Index Server are pretty much the last remaining ties between Microsoft and flat-file databases.

You mean SQL in the sense of "SQL Server" right? I somewhat doubt that they discontinue Access - it still fits a specific spot in the marketplace IMHO.

On flat-file databases: Actually the Registry is nothing but a big hierarchical database…

MFH said,
You mean SQL in the sense of "SQL Server" right? I somewhat doubt that they discontinue Access - it still fits a specific spot in the marketplace IMHO.…

Considering SQL Server Express is available for free, I imagine the next incarnation of Access will be a SQL Server variant under the hood.

Ok, not quite WinFS (which people don't get is essentially implemented using NTFS and search in Windows Vista/7 already.)

What is an easy way to describe this...

WinFS was a database sitting on NTFS used only for user content.
Protogon is a database that mimics/holds NTFS that can be used for anything.

So instead of a database sitting on a FS pretending to be a pseudo FS, Protogon is a database technology that actually holds what it pretends to be NTFS along with other data and features.

Think Windows Search using MSSQL, and then shoving NTFS data into a table that looks like NTFS to applications.

thenetavenger said,
Ok, not quite WinFS (which people don't get is essentially implemented using NTFS and search in Windows Vista/7 already.)

What is an easy way to describe this...

WinFS was a database sitting on NTFS used only for user content.
Protogon is a database that mimics/holds NTFS that can be used for anything.

So instead of a database sitting on a FS pretending to be a pseudo FS, Protogon is a database technology that actually holds what it pretends to be NTFS along with other data and features.

Think Windows Search using MSSQL, and then shoving NTFS data into a table that looks like NTFS to applications.


Then, as I've said before, it's more like WinFS' very early versions, such as OFS, since those were actual file systems in & of themselves.

MASTER260 said,

Then, as I've said before, it's more like WinFS' very early versions, such as OFS, since those were actual file systems in & of themselves.

Ya, sort of...

The thing with WinFS is that it was a set of technologies and concepts, and people conflate them.

The main overall goal was a searchable database like file system.

Which Vista and Win7 provide, easily, but don't a different FS or even a FS data store to achieve, because NTFS was able to track changes and do alternative streams and meta tagging, so it just needed a good indexing/search service with data added to what NTFS was already tracking for the content.

What we need is a technology that can be more like MSSQL, and so it with about the same level of overhead as NTFS and Windows Search. Which right now is possible in 'theory' but also is not easy to pull off. Microsoft will probably get there, even if not in Windows 8 and the new FS will be used just for Application Data Stores, and extending the Application virtualization technologies in Windows 7.

would love to see a database type system closer to the file system. I would say my only criticism of Windows 7 is the slow and unreliable search system. WDS 4 just doesn't seem to be able to keep up, I've looked into the architecture in the Win7 Rez book but there appears nothing to improve it in it's current state. (Poor performance, sometimes items cannot be found when they are there (unreliable) and it's slow at updating when files are added.

Apart from the search i don't have any complaints regarding the Filesystem in Windows NTFS is incredibly reliable and i think pretty quick. The only FS i know that can beat NTFS is ZFS but ZFS has a pretty large overhead in terms of memory and processing power. Ive yet to see an NTFS drive go into an unrecoverable state due to bad blocks or unsafe shut downs. The self healing in Win2k8R2 / Win7 is the icing on the cake, although again there are plenty of Win2k3 servers with hundred of thousands if not millions of files running without a problem.

I do love the Mac but the weakest point is the FS HFS+ is on it's last legs and needs to be improved or replaced. I have found it unreliable with lots of files and sometimes even suffering from the domino effect of corruption found on FAT32/FAT16 drives. I was hoping that 10.6/.7 would introduce ZFS or something new, but this is all off topic.

REM2000 said,
would love to see a database type system closer to the file system. I would say my only criticism of Windows 7 is the slow and unreliable search system. WDS 4 just doesn't seem to be able to keep up, I've looked into the architecture in the Win7 Rez book but there appears nothing to improve it in it's current state. (Poor performance, sometimes items cannot be found when they are there (unreliable) and it's slow at updating when files are added.

Apart from the search i don't have any complaints regarding the Filesystem in Windows NTFS is incredibly reliable and i think pretty quick. The only FS i know that can beat NTFS is ZFS but ZFS has a pretty large overhead in terms of memory and processing power. Ive yet to see an NTFS drive go into an unrecoverable state due to bad blocks or unsafe shut downs. The self healing in Win2k8R2 / Win7 is the icing on the cake, although again there are plenty of Win2k3 servers with hundred of thousands if not millions of files running without a problem.

I do love the Mac but the weakest point is the FS HFS+ is on it's last legs and needs to be improved or replaced. I have found it unreliable with lots of files and sometimes even suffering from the domino effect of corruption found on FAT32/FAT16 drives. I was hoping that 10.6/.7 would introduce ZFS or something new, but this is all off topic.

You are having issues getting Windows Search to keep up? Ok, break the index and rebuild it, something is wrong on your system.

(One thing I have seen people do is monkey with NTFS permissions, and turn off permissions on content, which keeps the engine from reading the files. So also reset your NTFS permissions if you have played with them, or have booted into Linux(etc), as it can monkey with permission when accessing NTFS volumes.)

Even on this laptop, I have 850,000 items indexed, including, voice, image OCR, and other search filters enabled. I can return anything in less than a second or two.

On my main desktop server, it has close to 2 millions items indexed, and works flawlessly and is fast.

As for ZFS, it could in 'theory' beat NTFS or 'hold' its own to NTFS with 'features', not necessarily performance. The remarkable aspect of NTFS is that it does have a dizzying level of features and still is one of the fastest FS technologies even with all the overhead of the features.

To mimic what NTFS inherently provides requires on Linux for example, creates a massive performance difference. Most FS technologies can hold their own with NTFS, but when you start adding on the services that provide the inherent functionality features of NTFS, it isn't even close.

Even Apple had a heck of a time getting just 'journaling' to work at an acceptable speed, which is why it was disabled in OS X for many years, and NT was doing journaling on a lot slower hardware in the 90s without a measurable performance hit.

ZFS is pretty much dead BTW. It also didn't fully meet the features of NTFS, with things like compression, etc. It did offer 'larger' sizes on a few things, but NTFS is already capable of hard drive volumes we won't see for another 10-20 years or more.

However, the non-Microsoft world needs something that can be like ZFS and provide the stability, functionality, and speed of NTFS. It is one of the holy grails of the OSS world, that hasn't happened yet, and Apple's FS technologies are far behind as well. They need copy-on-write, inherent encryption, journaling, transactions, compression, and the speed of NTFS all in a nice tight package.

Maybe Microsoft will allow or open up NTFS a bit more than they already have by allowing Apple and Linux read and write of NTFS. However without the Windows NT kernel and the FS layers that make the 'magic' happen with NTFS, it might be a moot point.

MFH said,
Weren't transactions introduced into NTFS (TxF)?

Yes, transactions came to NTFS with Vista iirc.

Anyways, I think this is a new FS probably for ARM devices that might work and act like the FS MS has on Windows Phone 7. Though I'm just guessing right now, only time will tell as we get more info.

GP007 said,

Yes, transactions came to NTFS with Vista iirc.

Anyways, I think this is a new FS probably for ARM devices that might work and act like the FS MS has on Windows Phone 7. Though I'm just guessing right now, only time will tell as we get more info.

NT History Lesson...

Transactions have been in NTFS far before Vista, it was a design goal of NTFS, and I believe was fully implemented in NT 3.51 back in 1993/1994.

Vista did introduce new developer level access to the NTFS transaction system, and adjusted the granularity, which is why it was noticed again with Vista.

NTFS has been transactional and using transactions for a long long time; however, prior to Vista it wasn't easy to access for developers(but possible). Windows NT itself (NT 3.51, NT 4, Win2K, WinXP) extensively used NTFS's transactional capabilities, which is one reason NTFS was solid and corruption pretty much had to be a hardware level failure to occur.

(NTFS's transaction features are one reason 'corrupt registry' and other things that were problematic on FAT and the Win9X OSes disappeared when people moved to Windows NT (Win2K,WinXP).

As for just for ARM, not so sure about that.

Also not so sure this test FS will ship as a main FS in Windows 8, it may be just used for application data store and expanded virtualization concepts. (Like Vista and Win7 already use to keep older software working and not interfering with other applications or data.)

It is a rather grand FS model, taking the best of the existing NTFS concepts and embedding them into a high performance database.

So whether this 'replaces' NTFS, comes down to how much of a performance gain vs overhead hit it yields, as the NTFS b+ tree index technology was good a long time ago, but showing some age with newer data storage indexing concepts that Microsoft has developed over the years. So there is room for improvement, but not if it requires a fairly heavy database engine that offsets the gains on modern hardware.

Apparently, all that testers need do is fire up Command Prompt, they should make sure and run it with administrator privileges, enter the following command: format DriveLetter: /FS:protogon /Q and hit enter.

DriveLetter should be replaced with the actual drive letter for the partition or drive that is being formatted with Protogon.

Testers that already created Protogon volumes reveal that the partitions and drives with the new file system appear to not only be faster compared to when formatted with NTFS, but also allow the usage of more space.

Hum said,
Apparently, all that testers need do is fire up Command Prompt, they should make sure and run it with administrator privileges, enter the following command: format DriveLetter: /FS:protogon /Q and hit enter.

DriveLetter should be replaced with the actual drive letter for the partition or drive that is being formatted with Protogon.

Testers that already created Protogon volumes reveal that the partitions and drives with the new file system appear to not only be faster compared to when formatted with NTFS, but also allow the usage of more space.


This works in the pirated builds. However, in the Dev Preview, when formatting, it just shows a, "Format failed," error twice... Actually, in the pirated builds, you don't even need to do /q

singularity0821 said,
I hope they will update that ugly disk space indicator

Don't know how much the could do (without changing it completly) but I guess just making it use the same colours as the disk space bars found on drives in the "Computer" window would be a welcome change.

singularity0821 said,
I hope they will update that ugly disk space indicator

And those Property Windows should be replaced with a UI consistent with Personalization Window.

singularity0821 said,
I hope they will update that ugly disk space indicator

Don't hold your breath. Updating that will take decades, just like with Task Manager.