What is WinFS?

[jjypk]

What if someone wants to upgrade to "Longhorn" and the person uses FAT32?

I use FAT32 myself and wouldn't want to waste time updating the file system to NTFS!

[uniacidz]

Can winFS also show the size of folders? this feature is missing from all windows OS's so far, and the few utilities that try to adress the issue appear to be quick hacks. This feature has been available to *nix users since stoneage :)

This what you mean?

[John Veteran]

What if someone wants to upgrade to "Longhorn" and the person uses FAT32?

I use FAT32 myself and wouldn't want to waste time updating the file system to NTFS!

NTFS is far better than FAT32. and longhorn will still be able to use fat/fat32 partitions, it just won't be able to format partitions with fat/fat32.

[Jean-Claude Van Damme]

Nor will you be able to run WinFS on FAT32.

[helmers]

Gnome Storage? Yeah that project has been started after there was first talk about WinFS. I wonder why they couldn't get the idea on their own, before MS.

Probably because they were busy working with other software. But what they do is basically give it away for free. Which I think is good. Also there has been a lot of debate about the actual benefits of such a system, compared to the memory requirements and system overhead.

[arcterex]

The problem is that of metadata. If winfs is going to pull information from files so that a query of "show me all mp3s by u2" it has to get the data from the metadata in the file. This works fine for mp3s and image files with exif data, and maybe avi or wmv files, but what about the rest? Also, what is ensureing that the metadata (in files that support it) is properly input? This is the same discussion that came up when gnome-storage was announced. Metadata searching is fine *if* the data is a) there and b) accurate. Unless WinFS is going to tag all my mp3s for me, and ensure that all the dates stored in the exif data on my pictures is correct of course :)

[pctuk]

Anyone used Avafind? Will it find files as fast and easily as this absolutely FANTASTIC program?

[helmers]

arcterex, what one may hope for is that since Windows holds such a large market share, the fact that more people will use the ID3 data(for example) will make people enter these data more carefully, and even though it won't help with old files, it will probably give us better data quality in the future. ;)

...welcome to neowin, btw!

[Jean-Claude Van Damme]

Probably because they were busy working with other software.

Right.

[Renfrew-guy]

Indexing has also been in Windows -for ages- in the form of Index Server.

WinFS expands the ability of the user to search for file system data regardless of location and obtain results very quickly. WinFS sits along side NTFS. WinFS is based on a SQL Server Store, which is not optimized for non relational data. Therefore WinFS, is good for relational data such as file metadata, but not to store files themselves ...... I would expect WinFS to become the File System (and thus store the files themselves also) in a later version of the Windows Client (or at least I would hope that this is the case). Every property of every file is synced into WinFS and there are a bunch of optimized stored procedures from what I can see that run on top, so that instead of waiting for Index Server to trawl through your entire file and directory hierarchy within the file system to find something, it is more or less found instantly from WinFS.

I think of WinFS (as far as the Windows File System is concerned) as an index to help you find what you're looking for quicker than you can today. Ideally, you would be able to say, "show me everything I've communicated with Aaron on in the last 30 days" and in a couple of secs, it would pull the information (thus links to corresponding files) from WinFS and present it to the user. A much richer experience than today. It doesn't seem that data from the MS Office suite is indexed in WinFS though at the moment. Maybe that will change before LH ships (whenever that is) .... who can say ....

This is based upon what I have read in MSDN and from the build I got at the PDC last year and lots of tinkering around.

[LispyGlitter2]

wow this guy is l337 :rolleyes: Any other hidden things you want to reveal to us?

[Renfrew-guy]

So ....... to summarize:

- WinFS sits alongside NTFS. It contains metadata from every file within NTFS to help you find what you're looking for much quicker than today

- WinFS is not a replacement for NTFS. Well not for the moment at least.

- WinFS does not contain files from NTFS

- The master write to a file, is still controlled by NTFS

- Updates to the metadata within WinFS is event driven and done on the fly when updates to the properties of a file within NTFS occur

Cheers

Scott

[goheels681]

With WinFS, what difference would the user actually SEE? If the structure of the folders actually different?

[Renfrew-guy]

The structure of the folders is not different.

WinFS deals with the search paradigm which will become (I feel it already has in fact) the next major headache of personal computing. When LH ships, the average PC is destined to have roughly a 6GHz CPU, 1TB+ disk space and around 4GB RAM. With devices such as the iPOD meaning that users rip their music collections to their PC, in addition to digital picture collections, and no doubt video over time as well as traditional file types ...... finding what you need is going to be like the needle in a haystack analogy.

WinFS will create a volume for each partition on your PC. In addition it will create a volume for Documents and Settings. It will index file metadata contents from each of these locations (not the Windows nor Program Files Directories). Attribute metadata is what is important. The ideal scenario is "Find me all files that I communicated with Paul on in the past 30 days" and it will find every picture, document, video, email .... etc in seconds. That presents a hugely more compelling search paradigm than is available today, but will be necessary with so much data on a PC.

The name Windows File System is misleading. I liked the original accronym better - Windows Future Storage. Don't think of WinFS as a file system - it's not. NTFS is a file system. Overtime (let's say the Blackcomb timeframe), I would expect WinFS to become the store for all data on the PC, but it will not become the file system itself. That's not to say that WinFS cannot contain files - it can. However(in the LH timeframe), it can only contain/subsume file which are relational in nature (e.g. an Outlook contact is purely and simply metadata). Such file types could be synchronised into WinFS and stored there (in contradiction to earlier posts). However, files which consist of BLOB file stream data will still be stored in WinFS - their attribute data will be stored in WinFS so that the file can be found easily, but the file itself (and the master write to it) will still be controlled by NTFS. The Yukon store (as I've said many times before) is not optimized for non-relational data, and so with the LH client, only relational file types can be stored there. However, metadata for any file in the file system (non-relational file .... e.g. .wma :-)) can be stored in WinFS, whilst the file lives in NTFS. Hopefully in the Blackcomb timeframe that will change, as I would hope that by that time Windows/WinFS will be based upon a Yukon+1 store which will be able to be optimized for relational and non-relational data. When this occurs, WinFS could become the store for all file types, however it would not be the file system. NTFS is very powerful and provides everything from quota management to bad sector remapping to compression and effective change control. Why reinvent the wheel? With WinFS and NTFS you get a marriage of mutually beneficial technologies.

Scott

Edited June 25, 2004 by Renfrew-guy

[McGazza]

Thanks for the article its cleared a lot up, its ahsame i didnt read it earlier :) :happy:

[jackslap]

And there are 3 or more incarnations of it that will be presented in differant stages of development to see what catches on and seems most userfreindly in the long run. Indications are the development will take 2 or 3 OS's before a final version.

[Renfrew-guy]

I agree that version 3 of a product (provided numbering didn't start at 3) is a good place to take stock and make sure that the project is set for success over the next 3-5 years. This period will cover A number of release cycles. By this stage, the product (earlier versions) will be in production at customer sites and their business-scenario-technology dependancies will drive the feedback cycle back into the product.

As for:

And there are 3 or more incarnations of it that will be presented in differant stages of development to see what catches on and seems most user freindly in the long run

Between this statement and the next it's not clear if you're saying that it will take 3 iterations/product versions of the WinFS store to have evolved and released before the will have a solid store/foundation to build on. I don't know where your getting the number 3 for product versions. The earliest builds of the LH client used the Shiloh (SQL 2000) store (=1st version), then the store was switched over to be a Yukon store (=2nd version). It stays with the Yukon store now, so I'm not sure where v 3 comes from.

As for the user friendly part - nothing fundamentally changes - users will still obtain information from the PC in the same way - ^^^primarily^^^ through the shell. There will be a Default Store created and one for Documents and Settings - consists of the My Documents folder hierarchy and it's content - essentially the data will live in WinFS. In addition a WinFS stored is created for every partition/volumen on every hard disk. The contents of each drive will be indexed (%windir% and %windir%\system32 are excluded from this search). All relational data types will automatically synchronized into WinFS (the File Promotion Manager - a service which runs within WinFS. The FPM is responsible firstly for ensuring that all metadata from all eligible file types/locations on the local machine gets synchronized into WinFS. Secondly, the FPM is resposible for managing Stored Procedure creation which will index each of the WinFS stores allowing users to find what they're looking for in seconds). I think that Microsoft WinFS dev need to look at auto-content archiving of data based upon last date access to keep storage costs and associated management as low as possible.

Files can be moved from WinFS to non-WinFS stores (e.g. the NTFS folder hierarchy). File promoters and demoters serve this purpose. Promoters move the file and it's contents and metadata into the appropriate WinFS store on the machine. When the file machine is moved to a non-WinFS store, the promoter (one required for each fiile type) will take the contents of the database record(s) concerns from WniFS, construct the new files with the correct extension, and move them into the non-WinFS store on the target machine.

Going back to the shell - it is being redesigned to be more task centric - an effort to help the un-assisted user is able to find the information they need in many quicker and more intuitive ways than those which are available today with Windows XP.

The user doesn't have to learn about relational database technologies in order to take advantage of WinFS - it will be largely seamless to the end user. Databae Reads and Writes performed against WinFS will be done in the back ground. The user simply has to interact with the shell as is the case today and all data will appear there. All file related metadata (music artist on a .wma file in the EXIF header for example) is sync'd into WinFS. A bunch of stored procedures run against the host

Indications are the development will take 2 or 3 OS's before a final version.

Incorrect. You're implying that until XP+2/3 (Longhorn, Blackcomb and then what ever comes next), it won't be until that time that a final version of WinFS will be within the Windows Product. Each Windows OS that ships with a WinFS store will be based upon the store from the previous version of SQL Server that shipped (=most current SQL DB version in the market place). The WinFS store which will ship in the box with the LH Client will be based upon a final RTM version of the Yukon (SQL Server 2005) DB store.

If you are saying that it will take Microsoft 3 goes (I presume you mean Windows OS versions) to get WinFS right, that would take close to 10 years according to the mean development cycle of the Windows platform. Also, getting something "right" implies that it is complete or absolute. WinFS will continue to change as new versions of the Windows platofrm are developed and released to the market place. Also, right is "subjective"- you could argue that for the Personal Assistant persona, documentation and all other essential required information (relational data types - OLE Compound Docs etc ....) is stored in WinFS and the new search options in the UI will be super easy to use and the user experience will be super fast. The PA has all she needs - **** In the red corner - round one does to WinFS ****. However, this would likely not be enough for a VP who requires additional functionality - perhaps accessing non-relational file types, providing storage for non-relational file types etc In addition, you have an ISV who wants to have WinFS index all copies of your Customer Connections - ACT database. A sync adapter has to be written to sync the information and related metadata from ACT into WinFS. A sync adapter is just an extension to the sync API. Every application which uses it's own store today for useful metadata like email (Notes, Eudora etc) will need to write a Sync Adaptor to sync this information into WinFS so that all critical information can be stored and searched on in one place - WinFS..... V1 in the LH Client, WinFS v 2 is currently slated for Blackcomb.

Bottom Line with WinFS. WinFS versions which ship in the box with the Windows Operating System will contain the SQL store from our current version of our SQL Server database in the market place. In the case of the Longhorn Client, Yukon (SQL Server 2005) will be the store version provided for use. No beta versions of WinFS wil ship in the box (with Windows) when a new version of Windows is released to the market place. The risk of instability is too high. Following the same logic, the Blackcomb Client will be based upon a Yukon+1 store version.

[Renfrew-guy]

Files can be moved from WinFS to non-WinFS stores (e.g. the NTFS folder hierarchy).? File promoters and demoters serve this purpose.? Promoters move the file and it's contents and metadata into the appropriate WinFS store on the machine.? When the file machine is moved to a non-WinFS store, the demoter (one required for each fiile type) will take the contents of the database record(s) concerns from WinFS, construct the new files with the correct extension, and move them into the non-WinFS store on the target machine.? ??

Please note that in the text above there is the following sentence:[/b]e:

(one required for each fiile type) will take the contents of the database record(s) concerned from WinFS, construct the new files with the correct extension, and move them into the non-WinFS store on the target machine.

Edited July 2, 2004 by Renfrew-guy

[Glowstick]

There will be a Default Store created and one for Documents and Settings - consists of the My Documents folder hierarchy and it's content.

There won't be a seperate store for Documents and Settings. My Documents and all are plain dynamic sets over all stores system wide. System bound data that will be stored in WinFS will go into the DefaultStore, which will also be the default location for the save dialog (unless the application overrides this, e.g. like for remembering the last location, like most applications do). Whether you store your documents in the DefaultStore or by mistake in the user created Pr0nStore or the WinFSStore of one of your additional harddrives doesn't matter, since the dynamic set turns it up no matter where it hangs out.

[jackslap]

Let me try that again, 3 differant versions of winfs, how differant I don't know, just that 3 versions were being developed, the plan being, to continue the development over an extended period 2or 3 OS's ,that may mean versions of longhorn alpha's they are each individual OS's or they may have meant full blown OS's these corporations look at time spans a little differantly than we do, they tend to plan thuings way down the road, so your guess would be as good as mine.

[Renfrew-guy]

There won't be a seperate store for Documents and Settings. My Documents and all are plain dynamic sets over all stores system wide. System bound data that will be stored in WinFS will go into the DefaultStore, which will also be the default location for the save dialog (unless the application overrides this, e.g. like for remembering the last location, like most applications do). Whether you store your documents in the DefaultStore or by mistake in the user created Pr0nStore or the WinFSStore of one of your additional harddrives doesn't matter, since the dynamic set turns it up no matter where it hangs out.

QUOTE (Renfrew-guy @ Jul 2 2004, 11:58) There will be a Default Store created and one for Documents and Settings - consists of the My Documents folder hierarchy and it's content.

Sorry for the error. I totally agree with your correction. It was the early hours when I began typing the mail (it's too damn hot here at the moment). I meant to put "There will be a DefaultStore volume created which will contain the My Documents content for each user"

It is interesting though that the majority of data it will subsume in my case will be music video and audio content which are all BLOB based and store their attributes in the EXIF header. WinFS can sync in the attributes for each of these files to allow for quick and easy search, but storage of the raw file stream is back in NTFS and the master write to that file and it's attributes will be controlled by NTFS.

Cheers

Scott

[Glowstick]

While the FileStream backend is NTFS, all requests will be tunneled thru the WinFS driver, so it has full control over the incoming data. Files will be accessed via a heavily modified and optimized NetBIOS stack. WinFS files will be addressed via UNC paths ( \\localhost\[folders...]\arbitraryfilename.ext ). If I remember it correctly, ACL's are also managed by WinFS, the NTFS filestream ACLs will be limited to the context WinFS runs on, means no fiddling by outside applications nor the administrator. Everything goes thru the modified NetBIOS interface.

While not claimed anywhere by MS, I wouldn't wonder, if the promoters can monitor write accesses to specific parts in the file. For instance if a fileformat stores all metadata in a fixed size header at the beginning of the file, the demoter can decide whether it kicks in or not after write accesses (speed optimization and stuff).

--edit--

On top of that, the promoter will probably be deferred to some time after the last write access too, otherwise you create pressure on the system and WinFS, for constantly promoting tiny changes, especially when an application is writing out a large filestream.

Edited July 2, 2004 by Tom Servo

[Renfrew-guy]

While the FileStream backend is NTFS, all requests will be tunneled thru the WinFS driver, so it has full control over the incoming data.

If I remember it correctly, ACL's are also managed by WinFS

That's an interesting one. If the application in question makes a call using WinFX then I would understand the request being tunneled through the WinFS driver. However, if a straightforward Win32 request is made remotely over RPC then I would think the request would not be tunneled, but would instead use the same mechanism as it does today?

I understand that the LAPI essentially sits abrest/over Win32 and calls can be made using WinFX to the system via csrss.exe, but, remote Win32 requests are going to have to be used remotely for some kind of file access potentially. I would understand the call coming through the WinFS driver to get to files in WinFS volumes - like those in My Docs (or should I say the Documents Volume). However, NTFS is still authoritative for writes to all files that reside on the NTFS file system which are semi structured/non-relational in nature - that includes the attributes that have been sync'd into WinFS.

WRT security, every object in WinFS has a DACL and this the standard Windows DACL model continues to be used. The Security Subsystem deals wiith all access/authorization requests in the same way as things work today.

Cheers

Scott

[Glowstick]

Actually, the current WinFX way to access filestream data is a byte array, not a System.IO.FileStream on the UNC path. However the NetBIOS interface Win32 applications used to read filestream data can enforce access control, and IIRC it will. But you know the deal, it's not yet beta, and I've been told lots of stuff has changed meanwhile.

Anyway, ACLs is the least concern for me. I actually want a GetFileStream() method on the WinFS filebacked item, to skip around this whole UNC path retrieving mess, that's currently there for managed/WinFX applications. Especially dealing with large data isn't exactly effective on a byte array. I forgot how it exactly worked, but deriving the UNC path from a WinFS File item wasn't exactly nice or elegant.

[jackslap]

If you atre really interested in see it in action have a look here:

<< removed >> - If you post a link, make sure it doesn't require membership.

Edited August 18, 2004 by gameguy