Windows not making use of my 4GB RAM.


Recommended Posts

hdood
No you can't. You can only allocate as much as you have addresses for in the virtual address table.

You can only map 2GB into the virtual address space at a time. You can still allocate more (both virtual and explicitly physical). What you say only applies if you're not willing to manually map chunks of allocated memory in and out.

It's not done much, but really, that's just because of how things worked out. If 32-bit systems with >4GB hadn't been superseded by 64-bit before they became common and relevant, and Windows had supported PAE, you would have seen more applications do this (and, I suppose, simplified APIs).

Link to post
Share on other sites
Brandon Live
As will OS X, FreeBSD, Solaris, and every other modern OS. This is purely a Windows issue, and boils down to the fact that the Windows world is closed-source. The open source world had the advantage that they could (for the most part) simply update the drivers to be PAE aware. No such luxury on Windows, where the drivers are binary and there's virtually zero chance of a vendor bothering to do anything after the product has launched. So how do you fix the compatibility issues? You introduce artificial limits in Windows.

First of all, no. OS X doesn't make use of PAE for extending the address table except in XServe. FreeBSD has a PAE kernel but the FreeBSD distribution does not support it, and most drivers don't work correctly with it.

I'm not aware of open source drivers for OS X, and many Linux and FreeBSD drivers are closed source.

Of course this isn't at all applicable to this situation since PAE wouldn't help any OS address more memory when the chipset doesn't support it.

Link to post
Share on other sites
Brandon Live
You can only map 2GB into the virtual address space at a time. You can still allocate more (both virtual and explicitly physical). What you say only applies if you're not willing to manually map chunks of allocated memory in and out.

It's not done much, but really, that's just because of how things worked out. If 32-bit systems with >4GB hadn't been superseded by 64-bit before they became common and relevant, and Windows had supported PAE, you would have seen more applications do this (and, I suppose, simplified APIs).

Windows supports PAE (for Server editions) and includes the Address Windowing Extensions APIs for making use of more than 2GB of memory from a 32-bit process. But very few if any applications make use of this, and any that do are certainly server applications not relevant to this thread.

Link to post
Share on other sites
hdood
First of all, no. OS X doesn't make use of PAE for extending the address table except in XServe.

So how does a Mac with 6GB work?

FreeBSD has a PAE kernel but the FreeBSD distribution does not support it, and most drivers don't work correctly with it.

The file server I have sitting in the next room running the official AMD64 distribution says otherwise. It is not 2006 anymore. It's true that it has had problems, but that just boils down to the fact that it's less popular than Linux and has less support, meaning things move slower.

I'm not aware of open source drivers for OS X, and many Linux and FreeBSD drivers are closed source.

"Many" is an exaggeration. Most are, and if we're going to talk about desktop users, the only thing people are going to come across that is binary-only is graphics and wireless drivers (if running Windows drivers).

As for OS X, you're right, it doesn't belong on the open source list (although it doesn't have the problems either).

Of course this isn't at all applicable to this situation since PAE wouldn't help any OS address more memory when the chipset doesn't support it.

Obviously.

Windows supports PAE (for Server editions)

I know it does, and maybe I should have qualified it with "client editions".

and includes the Address Windowing Extensions APIs for making use of more than 2GB of memory from a 32-bit process.

Yes. Like I said, you can allocate "any" amount of memory you want using things like AWE or even file mappings.

But very few if any applications make use of this, and any that do are certainly server applications not relevant to this thread.

Yes, but so? What I said was still correct. The fact that few use it is simply because it was made redundant before it could become relevant.

Link to post
Share on other sites
sammy2

My mistake, then why did win7 32bit release info said win7 32bit could utilize up to 32GB of ram ? while 64bit could use over 100GB...

Whats the point of that ?

Link to post
Share on other sites
Hawkeye666
Unless you run some really obscure software or hardware, the chances that you'll have any compatibility problems with 64-bit Windows are slim to none.

If you can get the 64-bit version for free, there is no reason why you shouldn't install it. Maybe you can save it until the next time you have to reinstall?

Hood is right, if you can get it go for it. 64bit is more stable and has extended CPU functions that allow it to manage memory such that multiple 32-bit apps will not be any where near as likely to crash each other. There are also memory advantages to using 64-bit. For instance each app has its own memory pool so it will have slightly more to use on its own.

As for 4GB in general go for it. The more you have the less that gets paged to disk. This makes Windows and apps faster when there is a reduction in the amount of paging from memory to swap file going on. Memory is cheap go for it.

Another misconception is that PAE will give you the full 4 GB with a 64 bit O/S. It will not if your BIOS and chipset do not both support remapping the memory address between 3 and 4 to the line above four. For instance the Intel 945 chipset will not access that memory regardless of the CPU or the O/S. Not all 64-bit CPU compatible Intel chipsets support this remapping. To my knowledge all AMD chipsets that support 64-bit CPUs do support it.

Link to post
Share on other sites
Brandon Live
So how does a Mac with 6GB work?

Using 64-bit addressing.

The file server I have sitting in the next room running the official AMD64 distribution says otherwise. It is not 2006 anymore. It's true that it has had problems, but that just boils down to the fact that it's less popular than Linux and has less support, meaning things move slower.

An AMD64 distribution doesn't use PAE. In long mode, AMD64 CPUs don't support PAE as an address translation scheme.

"Many" is an exaggeration. Most are, and if we're going to talk about desktop users, the only thing people are going to come across that is binary-only is graphics and wireless drivers (if running Windows drivers).

Windows drivers?

"Many" isn't an exaggeration when virtually everyone is running at least one closed source driver (since virtually all of the GPU drivers fall into this bucket).

As for OS X, you're right, it doesn't belong on the open source list (although it doesn't have the problems either).

Because it doesn't use PAE...

Yes. Like I said, you can allocate "any" amount of memory you want using things like AWE or even file mappings.

Again I think you're confusing terms. A memory mapped file uses address space... that's the whole point of a memory mapped file. If you write something to a file directly, that doesn't count as "memory allocation." That's just I/O.

Yes, but so? What I said was still correct. The fact that few use it is simply because it was made redundant before it could become relevant.

No. Few use it because it introduces a lot of complexity and overhead which nobody wants to deal with, and has so many limitations that what you're able to do with the memory you allocate via AWE that it's not even feasible to use it in most situations. Besides, it doesn't let you "allocate" memory that isn't mapped to address space. It really just extends the address space.

Hood is right, if you can get it go for it. 64bit is more stable and has extended CPU functions that allow it to manage memory such that multiple 32-bit apps will not be any where near as likely to crash each other. There are also memory advantages to using 64-bit. For instance each app has its own memory pool so it will have slightly more to use on its own.

Absolutely nothing in this paragraph is true.

Link to post
Share on other sites
neufuse
My mistake, then why did win7 32bit release info said win7 32bit could utilize up to 32GB of ram ? while 64bit could use over 100GB...

Whats the point of that ?

If you have PAE you can use up to that much in 32-bit.... but you get a performance drop using PAE in how it works....

Link to post
Share on other sites
Brandon Live
If you have PAE you can use up to that much in 32-bit.... but you get a performance drop using PAE in how it works....

Nope. Windows client editions will not use PAE address translation to address memory, so you'll never get more than 4GB in a 32-bit client edition of Windows. Only server editions support PAE for memory addressing. This is separate from operating the CPU in PAE mode, which 32-bit Windows always does regardless of edition (this is required for NX bit support and such).

Link to post
Share on other sites
hdood
Using 64-bit addressing.
Because it doesn't use PAE...

Source? Everything I can find seems to say that it does use PAE.

An AMD64 distribution doesn't use PAE. In long mode, AMD64 CPUs don't support PAE as an address translation scheme.

You are of course completely right. My mind drifted off into something unrelated and irrelevant.

Windows drivers?

Yes. It's fairly common to use them on Linux because the chipsets tend to be undocumented.

"Many" isn't an exaggeration when virtually everyone is running at least one closed source driver (since virtually all of the GPU drivers fall into this bucket).

Yes, the same driver. Even if ten million people are running an NVIDIA driver, it is still only one driver. Granted, it's an important driver, but it's still only one.

Again I think you're confusing terms. A memory mapped file uses address space... that's the whole point of a memory mapped file. If you write something to a file directly, that doesn't count as "memory allocation." That's just I/O.

No. Only things that are actually mapped into your address space use address space. You can create a mapping that points to virtual memory rather than a file, and then you can map this (or parts of it) in and out of your address space. The only space that is consumed when it is not mapped is the space it takes to store the handle. AWE does the same, except you are allocating physical memory to map in and out on demand.

No. Few use it because it introduces a lot of complexity and overhead which nobody wants to deal with

I disagree. People don't use it simply because 64-bit Windows made it redundant before it really became relevant.

Link to post
Share on other sites
PGHammer
Okay my system can handle 64-bit.

15092009002958.png

Well since I have 4GB of ram, won't it be very wasteful if I do not get the 64-bit OS instead? Even if I am not using professional software.

64-bit makes sense even if you don't have 4 GB of RAM; however, if you have that much (or more), as long as you don't have driver/application issues, why run 32-bit?

I helped three people switch to 64-bit last year, and I switched myself this past winter. None of the switchers (including me) had more than 2 GB of RAM at the time of the swiitch (my upgrade to 3 GB was from 1 GB, and was post-crossgrade).

If you are running Windows 7 and your hardware is x64-ready, then that is what you should be running, even with as little as the minimum amount of required RAM.

The wasteful part is not running 64-bit despite having full hardware and application support for doing so.

Link to post
Share on other sites
neufuse
Nope. Windows client editions will not use PAE address translation to address memory, so you'll never get more than 4GB in a 32-bit client edition of Windows. Only server editions support PAE for memory addressing. This is separate from operating the CPU in PAE mode, which 32-bit Windows always does regardless of edition (this is required for NX bit support and such).

I should of said server editions, wasn't thinking :)

Link to post
Share on other sites
Brandon Live
Source? Everything I can find seems to say that it does use PAE.

What is your source? All of Apple's documentation suggests that Tiger and later support a 64-bit physical address space via AMD64 and this is how greater than 4GB of physical memory can be used by the system. All of their machines that support >4GB of memory are 64-bit machines (the only 32-bit machines were the original Core Solo Macbooks and Mac Minis, neither of which can support more than 4GB of memory using their Lakeport chipsets). If you have some documentation suggesting they use PAE please provide it.

The lack of PAE mode was also Apple's explanation for the lack of NX bit support for 32-bit processes on OS X, by the way.

No. Only things that are actually mapped into your address space use address space. You can create a mapping that points to virtual memory rather than a file, and then you can map this (or parts of it) in and out of your address space. The only space that is consumed when it is not mapped is the space it takes to store the handle. AWE does the same, except you are allocating physical memory to map in and out on demand.

Which API are you referring to that lets you store data in virtual memory without mapping it to the process' address space? You can do this with physical memory via AWE, as AWE lets you override the memory manager's control of physical memory, creating views over physical memory inside the process' virtual address space. But the process can't access that memory without first mapping it to its own address space. Nobody uses this API because it basically requires you to re-implement the memory manager, and opens you up to making a real mess of things not just for yourself but for the whole system. I'm surprised this wasn't deprecated for 64-bit processes, actually.

Allocating "virtual memory" that isn't mapped to virtual addresses doesn't even make sense. The whole point of virtual memory is that it can be backed by any kind of storage and addressed the same way no matter what that backing storage is. If you meant you can write stuff to a file that isn't mapped to your address space, of course you can. But that's irrelevant to this topic. If by "virtual memory" you really meant the page file, the only way to access that is by using virtual memory (and thus some virtual address space).

I disagree. People don't use it simply because 64-bit Windows made it redundant before it really became relevant.

Not at all. It never would've gained traction. Nobody wants to implement their own virtual memory system.

Link to post
Share on other sites
hdood
What is your source?

I don't have any credible technical documentation about OS X at all, only blog-quality articles, so I'm certainly open to you being right. Do you have any documentation?

Which API are you referring to that lets you store data in virtual memory without mapping it to the process' address space? You can do this with physical memory via AWE, as AWE lets you override the memory manager's control of physical memory, creating views over physical memory inside the process' virtual address space. But the process can't access that memory without first mapping it to its own address space.

Nothing lets you store data without first mapping it. The point is that you can map it in and out as you want. Once it's mapped out, you can map something else in. You know this, so I'm not sure what this argument is about.

Allocating "virtual memory" that isn't mapped to virtual addresses doesn't even make sense. The whole point of virtual memory is that it can be backed by any kind of storage and addressed the same way no matter what that backing storage is. If you meant you can write stuff to a file that isn't mapped to your address space, of course you can. But that's irrelevant to this topic. If by "virtual memory" you really meant the page file, the only way to access that is by using virtual memory (and thus some virtual address space).

No, a file mapping can point to either a file or page file-backed virtual memory, even if the name implies otherwise. Once you've created the mapping, the memory is committed, but it isn't mapped. All you have is a handle, just as if it was a file. You then have to manually map it into your address space.

Writing a manager to handle large amounts of memory in chunks like this isn't that difficult, although it's obviously more work than just making the move to 64-bit, and so utterly pointless.

Not at all. It never would've gained traction. Nobody wants to implement their own virtual memory system.

It's not an issue of "want", if there is no alternative, it would have been used (and would likely have been simplified).

Link to post
Share on other sites
Brandon Live
Nothing lets you store data without first mapping it. The point is that you can map it in and out as you want. Once it's mapped out, you can map something else in. You know this, so I'm not sure what this argument is about.

Oh, well of course you can unmap something. But if it's no longer mapped, you can't do anything with it. It sounded like you were describing some "virtual memory" mechanism that didn't involve virtual addresses, which is what I was questioning.

No, a file mapping can point to either a file or page file-backed virtual memory, even if the name implies otherwise. Once you've created the mapping, the memory is committed, but it isn't mapped. All you have is a handle, just as if it was a file. You then have to manually map it into your address space.

It isn't virtual memory if it doesn't have a virtual address :) Page file != virtual memory. I believe what you're getting at is that you can reserve pages in the page file without allocating virtual memory. Is that what you meant? If so, you are correct, though I believe the practice is not particularly useful or common.

Writing a manager to handle large amounts of memory in chunks like this isn't that difficult, although it's obviously more work than just making the move to 64-bit, and so utterly pointless.

I still disagree with you there. The limitations are numerous and implementing AWE safely / reliably is non-trivial. Memory reserved via AWE is always physical memory and can't be paged to disk. It can't be made read-only, exectuable, or anything else that VirtualProtect controls. You can't have guard pages. That means you can't use that memory with APIs or frameworks that assume they can use VirtualProtect.

Link to post
Share on other sites
hdood
It isn't virtual memory if it doesn't have a virtual address :)

Well.. From whose perspective? It's not virtual memory to the process until it's mapped, but it is to the memory manager.

The actual purpose of this type of file mapping, I guess, is to let you have shared memory, since other processes can also open it. I.e you create the file mapping (createfilemapping) in the first process, map it and do something with it, then have the second process open it and do something to it. There's no rule that says you have to use it for that though. You can just as easily create one or more really big ones to use internally. The difference between doing that and using AWE is that you're allocating memory that is backed by the page file rather than just physical pages. You have to manage it manually, and some of the limitations you mention still apply, but I'm not claiming that it's a perfect solution, just that it is an option (even though you can probably count the people who've done it on one hand).

It's not that easy to predict what the world would have been like if 64-bit processors hadn't come along, but I'm just saying that if they hadn't, and large amounts of memory still did and became necessary, people would have had no choice. You can't just say that they wouldn't because it's more difficult. It's an absurd and unrealistic scenario really, but there you go.

Link to post
Share on other sites
BryanChung

Woah! I thought I closed the case but this goes on. But it's good everyone is sharing knowledge.

Link to post
Share on other sites
The_Decryptor
...

"Many" isn't an exaggeration when virtually everyone is running at least one closed source driver (since virtually all of the GPU drivers fall into this bucket).

...

Intel (the most popular GPU maker) have open source drivers and contribute new code back to the kernel.

Of course, Intel is the most "popular" maker because their integrated chips are in pretty much everything that doesn't have a proper graphics card, and it could be said that people using systems with integrated Intel cards probably aren't using Linux :p

Link to post
Share on other sites
Robotic
"Many" isn't an exaggeration when virtually everyone is running at least one closed source driver (since virtually all of the GPU drivers fall into this bucket).

Since there has been a lot of conflicting information on 32-bit and whether it's possible to use all 4GB or more, I had actually tried with modded kernels to allow more than 4GB of physical address space and hence the OS accesses the remapped memory above 4GB and uses all 4GB of my RAM. I tried this on Vista SP1 / SP2 and 7 build 7100 and have not seen any problems at all. This includes trying several driver versions for an nVidia GPU. Are you suggesting this is a fluke?

Also I keep seeing post abouts performance loss using PAE. Well I ran several benchmarks including graphics and did not see any change in performance. Isn't this because PAE is enabled by default on a 32-bit clients since XP SP2 and the advent of the non-executable bit and hence the OS is already using 64-bit PTEs?

"Beginning with Windows XP SP2, the 32-bit version of Windows uses one of the following:

* The no-execute page-protection (NX) processor feature as defined by AMD.

* The Execute Disable Bit (XD) feature as defined by Intel.

To use these processor features, the processor must be running in Physical Address Extension (PAE) mode. However, Windows will automatically enable PAE mode to support DEP. Users do not have to separately enable PAE by using the /PAE boot switch.

Or in other words

DEP is supported by Windows XP with SP2, Windows Server 2003 with SP1, Windows Vista, and later versions of Windows.

On 32-bit versions of Windows, hardware-enforced DEP requires PAE, which is supported by all Windows operating systems that support DEP. When DEP is enabled on a computer with a processor that supports hardware-enforced DEP, Windows automatically enables PAE and ignores the boot parameter values that disable it.

Wiki would seem to suggest PAE will be enabled by default for 32-bit XP SP2 and onwards with systems employing cpu's from sometime during and after the P4 era.

After AMD's decision to include this functionality in its AMD64 instruction set, Intel implemented a similar feature in x86 processors beginning with the Pentium 4 processors based on later iterations of the Prescott core.

The NX bit specifically refers to bit number 63 (i.e. the most significant bit) of a 64-bit entry in the page table. If this bit is set to 0, then code can be executed from that page; if set to 1, code cannot be executed from that page, and anything residing there is assumed to be data. Also note that it is used only with Physical Address Extension (PAE) page table format, because the x86's original 32-bit page table format obviously has no bit 63.

Now I would think that there are drivers out there that would cause problems but drivers can cause problems without PAE being involved at all, should we then suggest not using an OS because of it's possible instability with that driver? ;)

On a side note I personally haven't found a use for all 4GB whether using 32-bit or 64-bit so for me it is was an interesting but mute point really.

Link to post
Share on other sites
BryanChung

So if let's say I have an 8GB thumbdrive, does ReadyBoost helps in anyway? Must the thumbdrive be put into a USB2.0? :D Never really read about ReadyBoost before.

Edited by Bryan84
Link to post
Share on other sites
deactivated01032015

ReadyBoost is primarily meant for low RAM systems. So, if you have two or more gigabytes of space ReadyBoost won't help you much. On the other side, it's much better than using a Page File.

Link to post
Share on other sites
Brandon Live
ReadyBoost is primarily meant for low RAM systems. So, if you have two or more gigabytes of space ReadyBoost won't help you much. On the other side, it's much better than using a Page File.

ReadyBoost and the page file are completely orthogonal concepts.

Link to post
Share on other sites
BryanChung

I see =) So ReadyBoost won't seem to help much with my 4GB of RAM even though 3.12 is usable only. :p

Link to post
Share on other sites
Subject Delta
Woah! I thought I closed the case but this goes on. But it's good everyone is sharing knowledge.

You didn't really close the case because you listened to a bunch of FUD without listening to the people with real knowledge about X64 windows. The actual truth is that the overheads are small, not large as some people make out. With a modern system, barely even noticeable. Plus with X64 windows you get the added advantage of being able to use X64 software and game engines, which tend to be a bit faster than their 32 bit counterparts.

I run a lot of games on my rig, and I have yet to see a single piece of performance loss, all of my games run at equal speed to what they would in XP or Vista 32 bit.

Link to post
Share on other sites
This topic is now closed to further replies.
  • Recently Browsing   0 members

    No registered users viewing this page.