Running backups overnight, the backup program slowed and finally quit ("program not responding..."). A little bit of experimenting seemed to show that any program with high disk I/O exhibited similar behavior.

The first attached screen shot seems to point toward the cause of the problem, but I am beyond what little bit of expertise I have in these matters.The shot was taken copying C:-->M:, but the failure actually occurred while copying F:-->L: which would suggest that it isn't a problem with a specific drive.

Is this as serious as it looks? MEMTEST.EXE showed zero errors after running for an hour.

The second screen shot shows what things look like when the backup application stalls. It looked pretty much like this for about five minutes before I captured it.

Right-click on the picture and select "View Image" to make it more legible.

Does anybody know what is going on here?

tanstaafl.

Depends on what you think it looks like.

It does not mean that your memory is faulty. In this context, "fault" means that the information that was requested was expected to be in RAM, but wasn't, because it had been paged out to virtual memory on disk.

It does mean, however, that you have less RAM than the processes you're running concurrently need. It still works, nominally, because the OS is able to move data from RAM to disk when it needs more RAM for something else, but a hard drive is many orders of magnitude slower than your RAM.

This is normally not a big deal when you have multiple programs running concurrently when they're not really doing that much all at the same time. (For example, if you have Photoshop open with a huge image, it's okay for it to page all that memory out to disk when it's not being used, because you're doing something else. When you get back to Photoshop, it'll just pull it back into RAM and page out what you had been working with.) But when processes all actively running at the same time need more RAM than you have, it starts using the hard drive as intensively as it uses the physical RAM. Or tries to. And it's incredibly slow.

So: run fewer programs or buy more RAM.

Yeah, what Bitt said. "Fault" doesn't mean faulty in this case. It just means that there was a ram<->disk swap happening, which happens all the time on computers, even when things are working normally.




He's got the maximum possible memory for a 32-bit system, and this is an overnight backup (i.e., not a lot of other programs running).

A backup program should not cause a machine to run out of RAM and slow to a halt. Something else is going wrong here, something OS-related or software-related.

Sometimes I've seen AV programs get into race conditions with backup programs.

I've also seen bad disk drives cause that kind of behavior: Disk I/O slowing to a crawl and causing programs to stop responding.

Let's see what this tool has to say about the state of your hard disks, please?

Memory-mapped I/O (i.e. MapViewOfFile, which is the Win32 spelling of "mmap") probably also counts towards page faults. 300 page faults per second isn't very many, though; pages are 4Kbytes so that's only 1.2Mbytes/second. Is that about how fast the backup is going?

Peter

There is a memory problem, but with my memory, in that I had this problem, fixed it, and don't remember the details.

One problem I did have that could have caused this is the transfer mode, for some reason Winders like to disable DMA and use PIO, which is slow and buggy. If it's applicable for your version of windows, check that.

That's what was confusing me. I can't imagine how you're running out of ~3.5GB of RAM when you're not even using the computer. At the very least, I'd imagine that a decent backup program would give you the option to use fewer system resources.

What antivirus are you running, Doug? Personally I prefer MSE as it stays out of the way better than Norton or McAfee do. Then again, I think I'm forgetting that we had an antivirus discussion a little while ago, didn't we? Hmm...

About 2.7GB (if I'm reading it right -- it's quite small) is being used for caching. This is expected; Windows Vista (and 7) aggressively use memory for caching.

And, as Peter says, those page faults could well be caused by memory-mapped I/O. This is perfectly normal.

Right-click, View Image

You've got the memory and nobody is using it so the OS might as well use it to cache files. If an application does need that memory then it'll get released and allocated.

Oooh. He's got so much RAM that no one process can map it all (unless both the sysadmin and the process jump through hoops). Perhaps the backup program is sizing its buffers by system memory size, and running out of address space in MapViewOfFile? Is it possible to tell Windows to boot with only 2GB of RAM and see whether the problem goes away?

Alternatively, take a look in the Processes tab in Task Manager and see if the "VM Size" of the backup program grows endlessly, until it reaches 2GB and things start failing.

Peter

I notice in the images, the minimum page file size is only slightly larger than the amount of ram being currently used.

I know this isn't XP, but Windows XP was always a little flaky when it was allowed to manage the page file size. My system would always grind to a halt when XP decided to expand the page file size. The fix was to make the page file bigger, and make the min and max sizes the same.

Do later versions of Windows suffer the same problem?

Here are pictures of source and destination drives.

I think the hardware is OK, but there is something very flaky going on with the computer as a whole. Right now it works for about a minute, then freezes: no mouse, no keyboard, and when the diagnostics are on screen, no disk activity, no memory activity, then it will start working again. It has frozen twice in the process of typing this post.

Neither one of my backup programs will copy more than a handful of files before it freezes up, yet Windows Explorer copied 90 GB of files at an average rate of around 50 MB/sec without any problems at all.

I ran an 8-hour MemTest with zero errors.

I can't deal with this anymore tonight, I'm going to shut it off, and hope it works better in the morning.

tanstaafl.

Your D: drive error rates are less than healthy.

Download SeaTools and see what it reports. (It appears to be a Maxtor drive, and Seagate owns Maxtor now, and SeaTools claims to support Maxtor drives. If it's not a Maxtor drive, try and find a diagnostic utility from whoever the vendor really is.)

Or just boot a live linux CD, and run "smartctl -a /dev/sda" on it.

Not too much, as it turns out...

tanstaafl.

Are your IDE/SATA controller drivers properly updated on that system?

Apparently...

There are some outdated drivers, but in every case they are for equipment no longer installed, or for equipment that is (IMHO) operating properly so why go looking for trouble?

On a related note... what do I do to convince my computer that the HP ink jet printer, the on-board sound, and some other things as well, are no longer connected and never will be again? Apparently the drivers for them are still kicking around, confusing my Driver Detective program.

tanstaafl.

The HP printer drivers are probably in the add/remove programs screen somewhere, waiting for you to uninstall them.

If the onboard sound is disabled in the system BIOS but is still showing up in the device manager, then deinstall the drivers from the device manager via a right click.

If your only complaint is that these things show up in the Driver Detective screen: Fugheddaboudit. Those files don't go "active" until you plug in the device in question, and having the files sitting there shouldn't hurt anything. In what way, exactly, are they "confusing" driver detective?

Only in that they show up as outdated drivers, prompting me to do something about them. I guess the only real "confusion" there is mine. I just like things neat and tidy, is all.

tanstaafl.

Can you provide a teensy bit more detail, Mark?

I downloaded Knoppix 6.2 Live Linux (691 MB), burned it onto a DVD disk .iso, changed my boot sequence in the BIOS, and sure enough, the computer booted into Linux.

I found a terminal emulator icon, got to a $ prompt and keyed in smartctl -a /dev/sda and pressed the Enter key, only to find "smartctl not found" or words to that effect.

My CD tray wouldn't open under Linux (driver?) so I had to restart back into the BIOS to change the boot sequence again in order to get back into Windows and the bbs.

Surprisingly enough, I used to be marginally competent in Linux [Xenix, actually], but that was about 30 years ago. What do I need to do now?

tanstaafl.

Not bad. Perhaps Knoppix 6.2 is missing the smartctl command?
Try this from the terminal window. Prefix everything with sudo to get full root privileges:

sudo apt-get install smartmontools

Then try again:

sudo smartctl -a /dev/sda

## sda is the "first" drive, sdb the second, and so on..

Not the driver, but rather a facist software geek. They like to "lock" the drive door while the drive is in-use. Such as when running the operating system from it.

More likely, it was in /sbin or /usr/sbin, and the default Knoppix user probably doesn't have those in its PATH.

Also, are we sure that it's going to show up as an sd? It's almost certainly not going to be sda.

That said, what it smartctl going to show us that we can't already find in the Windows SMART utility?

Maybe.

My computer is deteriorating rapidly, to the point where trying to copy more than a few megabytes from one drive to another, any drive to any other (there are six hard drives in the system: 4 internal, 2 external) causes a lockup. All I/O activity ceases for about 40 seconds, then I can hear the heads do a rather noisy seek, the CPU activity spikes to 100%, and then things will resume for another two or three minutes. Unless (as frequently happens) the program doing the transfer times out with a "Program not responding" error.

Two different backup programs (Microsoft SyncToy, and Karen's Replicator) exhibit this behavior, as does my disk defrag program (Auslogics), as does Windows Explorer.

Occasionally I get the 40-second lockup even when not engaged in heavy disk I/O.

Unless someone can come up with a solution, I may have to Nuke and Repave. I do not want to do that - my "Control Panel --> Programs & Features" has well over 100 items in it, and that's after I've gone through and uninstalled everything I don't use. If I have to, then I will upgrade to Windows 7, 64-bit, which means learning a new operating system. And no, Linux isn't an option for me, of that 100+ selection in Control Panel, I suspect that the majority don't have Linux analogs.

Right now my backups are a mess, since the problem began part way through the backup procedure. I've run these backups many times in the past without problems, and I haven't changed anything other than to add more data to be backed up. Fortunately none of my data is on the system (i.e., C:) drive, just all the applications. My data on the D: drive is solid, my 1-TB F: drive not so much.

The fact that I/O activity on ANY drive causes lockup suggests that there is no single drive causing the problems. Or does it? I guess that if any one drive was doing it, it would be the System or C: drive. Do you see anything scary in the attached picture?

I've been fighting this for two days now and am beginning to become discouraged. I really don't know if it is hardware or software causing the problem.

tanstaafl.

edit: It's not hardware. I should have thought of this before... I went to a DOS prompt and used XCOPY to copy files. It is copying as I write this. It has copied 10 GB without a problem and is still going strong, looks like it is averaging about 60 MB/Sec. Now up to 16 GB copied. 20+ GB as I submit the post.

So, what is borking my disk I/O in Windows, but letting it run full speed in a DOS window?

edit #2 FOUND IT. (I think...)

It looks as though any time I have my Vantec external dock powered up with a hard drive in it the disk I/O for the entire computer goes teats up. Two different hard drives cause nothing but trouble. When the dock is powered down, everything runs like it is supposed to. Of course, since the dock is my major backup tool, it was always on when I was trying to figure this mess out. I strongly suspect that the dock is the problem, not the hard drives themselves.

It's the dock. Or maybe the SATA cable. Hey, maybe the adapter that the cable from the dock plugs into. Maybe the cable between the adapter and the motherboard. Maybe the SATA plug on the motherboard. Tomorrow I'll open it up and re-seat the cable where it plugs into the motherboard, see if that helps. I am not optimistic.

Hmmm... there's also a USB connector on the dock. If it fails to work on USB, that will be a good indication that it is the dock itself, and not any of the other possible things. And if it does work on USB, I'll just run it that way instead of SATA. It doesn't matter if it's a little slower, I run the backups overnight anyway.

edit #3 Yes, works just fine on USB.


db

Out of curiosity, what motherboard or chipset do you have? A screenshot of Device Manager with the System section may help identify it (I think, don't have Windows handy at the moment to reference).

And does the external Vantec dock plug into eSATA directly into the motherboard, or into some sort of eSATA PCI card?

Neither, actually.

There is an adapter plate that goes where a card would be fastened to the back of the computer case, but it does not go go into a motherboard slot. Instead it connects by cable directly to a SATA connector on the motherboard. The adapter plate is just a means of getting the SATA connection from the dock into the inside of the computer case, there are no electronics or anything.

tanstaafl.

As far as how you get your eSATA ports, thats the same trick I use on my Mac Pro, little adaptor plate with cables running into the system to 2 unused SATA ports. Works great for the few times I've had eSATA drives attached. They don't get much use though as I prefer my NAS instead.

I can't tell what nForce chipset you have, but it looks like you do have one based on the NVidia reference there. Main reason I asked is that the I/O degradation sounds eerily similar to the failures I've seen at two companies using NForce powered Dell XPS machines. The difference here is the Dell's all used NVidia Intel chipsets, not AMD. The reason for the failure was a bad system design that dumped heat from the CPU directly onto the chipset. As the system aged when being used to compile code all day or sit in an office overnight without air conditioning, the chipset would just start acting oddly, usually slowing or delaying IO to the point of the machine being useless.

I can't say for certain this is similar to your problem, especially since it went away when you stopped using the eSATA port. I'd still recommend double checking the chipset though internally, make sure it gets enough cooling, and doesn't have a ton of dust and such on it. To identify the main chipset, it's going to be a component on the motherboard that has a heatsink on it, usually located somewhere near the CPU or memory. The heatsink may have it's own small fan, though the heatsink/fan setup will be much smaller then the CPU one.

Here is a more useful screenshot.

tanstaafl.

Yep, the second shot of device manager confirms the NForce chipset is running your SATA ports too. Some motherboards have other storage controllers embedded, though this tends to be for extra ports above what a chipset can support, or for adding in newer standards that haven't been universally adopted yet in motherboard chipsets.