This is a story that should really tell us something broader about how computers work and how software that controls computers works:
- Often in weird ways.
- Often in totally unexpected ways.
- Often in sadistic ways.
- Often in seemingly nonsensical ways.
When trying to explain a complex technical concept I always try to start by making a real world analogy; I think it greatly helps understanding the nature of the concept I am trying to explain, which is the prerequisite to understanding the details.
So imagine you go to a restaurant and you order a bowl of soup. The waiter, along with the soup, brings an impressive selection of spoons, so that you can pick the one that better suits your eating habits and preferences. You are really hungry and in a hurry, so you grab a really huge spoon and finish off the soup in a snap.
The next day you enter another restaurant, and you order their specialty soup. After a while the waiter arrives with your soup and brings along a small luxury spoon, children’s size. You stare at the spoon with bewilderment and ask for a bigger one.
"I am sorry sir, I am afraid these are the only spoons we have. They are selected by our chef so that customers can properly enjoy their soup".
At first you are shocked; then puzzled; then... you name it. Anyway you decide to give it a go. The soup is excellent and the little spoon can be made to work.
Now imagine you go to a third soup restaurant and the chef mandates that clients eat their soup using a spoon so small it can only hold a single drop of liquid.
What excuse can the chef offer you that could possibly justify what you have to go through in order to eat your soup using such a spoon? I don’t think I would fall for anything.
End of analogy.
In the example the restaurants represent the various versions of Windows running on different hardware. The soup represents "data", while you get to be on the device driver seat. The chef is the Windows kernel and the spoon is the size of the "bucket" that you can use to move data from your device.
Let’s get a bit more technical.
The Windows kernel has a concept called "Map Registers". Map registers translate to the size of the maximum DMA transfer that you are allowed to program on your device.
Instead of confusing you with "Map Registers" I will be talking about Maximum DMA Transfer or use the simpler term of "DMA Limit" that we use in Unibrain’s Firewire software documentation.
The DMA Limit is of course decided upon by the Windows kernel.
- On x86 Windows running on plain-vanilla x86 hardware, the DMA limit is 2GB which means that... there is NO LIMIT. No DMA Limit. Any driver can program a DMA transfer as big as its heart desires, like 100MB for example.
- On x64 Windows running on x64 hardware the DMA limit is 1MB. This is not a typo. It actually says "One Mega Byte".
Before getting too upset about "Why didn’t Microsoft set a bigger DMA Limit on x64?" I think it is better that you consider a different question: "Why didn’t Microsoft set ANY DMA Limit on x86?"
In my opinion this is the point where Microsoft initially got it wrong. By not putting a DMA limit on x86 Windows Microsoft made the lives of driver developers much, much easier, and thus they achieved having x86 Windows drivers for everything under the sun.
The downside was that many of these drivers were of dubious quality, allowing applications to setup even 100MB DMA transfers (which is a huge waste of system resources).
If Microsoft had put a reasonable DMA limit into x86 Windows, like say 10MB, then driver developers would be really used to the idea of working with a DMA limit and then the transition to x64 would be much smoother.
Instead Microsoft adopted a different approach for x64. By setting the DMA limit to 1MB they are essentially telling driver makers:
If you can’t operate your device using max 1MB DMA transfers then we don’t want your driver running on x64 systems.
Fair enough I say; 1MB is more than enough and keep in mind that it is not a 1MB overall total. It is 1MB per DMA Adapter Object. If your device has multiple execution contexts, then it uses 1 DMA adapter object for each context, and you get a 1MB maximum per execution context.
Trust me when I say that 1MB IS ENOUGH to serve even the demanding 800Mbps of Firewire.
However, the fact remains that this change in the DMA limit from the x86 world to the x64 world was a bit dramatic. And it is the core reason why YOU are having so much trouble finding x64 drivers for the hardware stuff you have on your Windows PC.
I hope I haven’t bored you to death with all the above but this was the simplest introduction I could come up for the REAL ISSUE of this article.
You might have noticed that something is missing in my earlier description about how Windows sets the DMA limit: Unlimited on x86 OS on plain-vanilla x86 hardware, 1MB on x64 OS on x64 hardware.
So what about x86 OS running on non plain-vanilla x86 hardware?
What about x86 OS running on x64 hardware, which is the norm these days?
The common denominator here is that we are talking about hardware that can address more that 32 bits of PCI space. This feature was referred to as PAE (Physical Address Extensions) on those exotic x86 machines that first supported it, and it still appears that way on your x86 Windows running on your x64 PC on the Computer Properties dialog.
On these systems, there is no DMA limit as long as the total amount of PCI-addressable memory space is less than or equal to 4GB.
What’s the difference between "memory" (RAM) and "memory space"?
Well, memory space is a space where memory lives. Master of the obvious, am I not?
The PCI-addressable memory space is where you RAM lives.
Is there anything else inhabiting that space?
The answer is YES: Device memory.
Device registers and the device’s PCI configuration space are "memory mapped" (mostly), which means that they have PCI-level physical addresses that the operating system can use to access them. Your graphics adapter memory goes in there too.
A crucial thing that happens a little after you start up your PC is that the infamous chipset, working together with the BIOS and other mystical powers, reserves 0.5GB up to 1GB of PCI-addressable memory space for device memory. In most systems I have tested it was either 0.75GB or 1GB. I’ll refer to this as DeviceMemory.
Now let me reiterate one of the previous paragraphs and see if it starts ringing any bells:
On these systems, there is no DMA limit as long as the total amount of PCI addressable memory space is less than or equal to 4GB.
Ladies and gents, we have arrived at our destination.
If you bought 4 or more GB of RAM for your PC, running an x86 version of Windows on the aforementioned hardware then RAM+DeviceMemory>4GB, so one of two things will happen:
- The x86 OS will only "see" 3GB, or 3.25GB or 3.5GB of RAM.
Actually it will see VisibleRAM=(4GB minus DeviceMemory).
- More usually however, especially when modern x64 hardware is involved, the x86 OS will take advantage of PAE support on the hardware and then VisibleRAM=(Full RAM).
If the first option is what happened to you, then you are a lucky bustard (typo intended – I don’t use slang in my posts ;-)). The reason that you are lucky is that in your case the x86 Windows kernel continues to operate in the generous NO DMA LIMIT mode. Every driver on the planet is happy in that land.
If the second option is what happens to you, then you are in for a nasty surprise which is the article’s punchline:
On x86 Windows with VisibleRAM+DeviceMemory>4GB the DMA Limit is 64KB.
Once again there is no typo: SIXTY FOUR KILOBYTES
Or 16 map registers if you prefer.
So you installed 4GB of RAM on your x86 Windows PC running on x64 hardware and suddenly:
- Logοn takes several minutes
- File copies are slower than usual
- Network is sooooo slow
- Everything feels sluggish and defunct
There is a way out and it is pretty simple: edit your BOOT.INI and add /MAXMEM=3072 (or the equivalent with my beloved BCDEDIT on zVista). See if it makes any difference and then try readjusting it up or down until you find the maximum value that does not invoke the 64KB death penalty.
This 64KB DMA limit is the tiny spoon I was talking about in my analogy at the beginning.
Just try serving up a bandwidth hungry 800Mbps Firewire device using a 64KB spoon. A huge and coordinated effort is required in the code and then it will only work up to a point. After that point the Firewire device (usually a high res high fps digital camera) will be shooting data at the PC so fast that with a 64KB spoon you are just out of luck (data flies on the cable and your spoon is not there to receive it).
I don’t know how or why this 64KB value was decided upon. This is an interesting topic for speculation. Some said it was historical issues dating from the EISA bus or something. I personally can’t figure it out. It just can’t be stupidity. There must be some reason behind that. Or may be indeed it comes from the EISA days and they just forgot about it.
I would be very interested if you sent me some comments with your speculations on this.
So, it’s high time we brought this to a closing:
How can you tell if you are infected (other than by witnessing sluggish behavior)?
Well, to the best of my knowledge, you can only get this information from a kernel driver. There is a considerable chance that Windows does not use one single value as the DMA limit for all devices. May be they are using different values depending on the type of bus the device is found on, or even the type of device, so there might be no global value to report.
Based on my discussions on NTDEV and private emails, it seems like it is a per-bus setting, which means that all PCI devices probably get the same value.
Doing a SysInternals-like project to display this value on your system, with a self registering driver hiding in its resources, etc, is interesting, it requires several hours of careful work, but is otherwise trivial for a DDD. Did I forgot to mention the endless support afterwards?
So I am not going to do it. Or should I put it better, I am not going to redo it, since I have already done it somewhere else.
Seeing the DMA Limit from userland is a feature that I have added to Unibrain’s Firewire drivers (ubCorePro 5.21) after the technical investigation of this issue came to a close. If your motherboard has a 1394 port or you happen to have a Firewire adapter you can use our software to see the DMA Limit.
This is a f.r-e^e download from www.unibrain.com, mostly harmless to install :-)
Just make sure you download the Pro version.
Install ubCorePro, reboot, login and wait until Windows starts up all the drivers, then run FireCommander and type the DMALIMIT command to see the DMA limit that your system granted to our drivers.
Some words of caution for those who will not bother reading the manuals:
- You don’t have to uninstall ubCore in order to get back to using the Microsoft 1394 drivers. We have a tool called UBSWITCH that does exactly that, in both directions.
- There is no significant performance difference between the Microsoft SBP2 driver and our own. Don’t bother trying to see if you will gain anything on that aspect. Remember we are focused on industrial applications not end users. Industrial applications don’t use SBP2 so there is no incentive for us to try to make the driver faster than the MS driver (supposing SBP2 devices can go any faster).
Finally, if you care for the technical discussions of this issue as they took place on OSR Online, you can go to the NTDEV forum, select advanced search and search for "Maximum 16 map registers" without the quotes. You should see my name on the post that started the thread.
I have also posted a technical explanation for FireAPI developers on our informal newsgroup.
Have Fun!
Dimitris Staikos
