I've been tasked with expanding the list Windows platforms that our software is "officially qualified" to run on. Currently, it's 32-bit XP, and 32/64-bit Windows 7. I have to add to the mix, Server 2003, Server 2008, and Vista, in all available 32- and 64-bit flavours, with a mix of different editions (professional, enterprise, ultimate, etc) and service packs.

That adds a significant amount of testing that we have to do (5 release branches * 2 architectures * 25 operating system flavours * 1.5 hour and growing test suite) in a 6-8 hour overnight window of time. If I get my math right, that means worst case we have to have ~62 VMs simultaneously running our test suite, which, naturally, is bandwidth and IO intensive. Obviously, any way we can cut this down would be a great boon.

The most obvious is to skip all the different editions. Can anyone say with certainty if different Windows editions behave identically? For example, if I test on 32-bit Windows 7 Professional (which would be the lowest supported edition), do I really need to test on 32-bit Windows 7 Ultimate? Most of the differences between editions appear to be things like networking services, and bundled applications, which wouldn't affect our product, but are there kernel/dll differences between editions, as well?

Unless your product directly interacts with the features that are different in the different Windows editions, then your product will not behave differently.

Reducing it down to the lowest supported edition is a great idea.

Don't know what your product does though.

It's a multi-threaded video encode/decode library.

With the exception of some of the low level thread stuff, it's pretty much all cross-platform code. The trouble that we've noticed, and got us off on this direction, is that different Windows kernels seem to handle multi-threaded application exits different. Windows 7, for example, seems to kill all the threads, then start mopping up memory. Same with XP 32-bit. But XP 64-bit, however, seems to start mopping up memory before killing threads, and, consequently, yanking memory out from under threads that are still using it, causing crashes.

So that's where I'm hitting the known unknowns. I don't know if there are low level differences like that, based on edition. Looking at the Wikipedia list of Windows versions, I see notes for different kernels next to some editions (specifically the Enterprise edition), which makes me question if there are other kernel changes that aren't documented.

Do you have a better source of edition differences? I've looked at Wikipedia, and Microsoft's site, and the Microsoft data I can find is, unsurprisingly, predominantly high-level marketing overview of differences.

XP 64 bit was an odd OS. It was actually Server 2003 (NT 5.2), down to the kernel. I wonder if the different behavior you see is some difference between the "server" and "client" kernels.

That interesting. We do get the same crash-on-exit behaviour on Server 2003 and Server 2008.

Wikipedia has a bit more info including a few links about the kernel. It also reminded me that there were other "odd" 64 bit XP versions:

Windows XP 64-Bit Edition for Itanium systems, Version 2002 - based on the XP codebase
Windows XP 64-Bit Edition, Version 2003 - Itanium as well, built off the Server 2003 codebase
Windows XP x64 edition - The version you are seeing, Server 2003 derived for 64 bit x86 processors

At the time, Microsoft bought into the whole Intel story that Itanium would be the PC migration path to 64 bit. They built up a big separation layer (WoW64 - Windows 32-bit On Windows 64-bit), and all of it just carried over into the x64 versions, even for Vista and 7.

Back to your kernel issues, I find it odd you see differences between Server 2008 and Vista. Those should be lined up code wise when the latest service packs are applied (or so I thought). And Server 2008 R2 is supposed to line up with Windows 7. Any odd differences in Windows 7 32 bit vs 64 bit that you are seeing in the testing? You could verify the actual kernel file is identical (C:\Windows\System32\ntoskrnl.exe) between Windows 7 64 bit and Windows Server 2008 R2. May not reveal subtle changes in how they run when in server or client mode, but would at least identify if they are the same binary.

Fortunately, we get to ignore the Itanium junk.


That's what I thought, too. FWIW, I've not actually tested any of the Server versions, or Vista. Since we don't support them (yet), we don't have any licenses. The crashing behaviour is being reported by another group who released a product (depending on our API) and insists it has to be supported on all these other systems. The crash analysis was done by another member of my group, on a borrowed system. Oh, and to make things worse, which service pack is installed is yet another column on my test matrix... it's not sufficient to support a configuration having only the most recent service pack.


None that I've heard of. Mind you, we don't actually test on a Windows 7 32-bit system... we currently just test the 32-bit binaries on a Windows 7 64-bit machine.

We have some holes in our configuration test coverage, so my question is really more "is there a minimal set of plugs I can use such that all the holes are filled." I know the maximal set of plugs, but I'm trying to avoid using two full racks just for testing a DLL.

That's a good idea. I can try that. I can probably also check what libs get linked into our API, and see where they differ, as well. I think we do actually have one machine with 2008 R2 on it.