I have tried three or four syncs lately of a relatively significant size, and have had this problem appear at various points throughout. The disk integrity check goes fine, the files begin uploading, and eventually I get this Emplode behavior:



And the following on the serial port:

Unable to handle kernel paging request at virtual address c1000000

memmap = C0D68000, pgd = c0d68000

*pgd = 00000000, *pmd = 00000000

Internal error: Oops: 2

CPU: 0

pc : [<c004550c>]    lr : [<c00454ec>]

sp : c09a5e18  ip : 00000001  fp : 00000000

r10: 00492058  r9 : c08ae120  r8 : c09a5e30

r7 : 00000249  r6 : c0fff600  r5 : 00000057  r4 : c09a5e30

r3 : c0fff6dc  r2 : 00000000  r1 : 00002000  r0 : 00000057

Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  Segment user

Control: C0D6917D  Table: C0D6917D  DAC: 00000015

Process player (pid: 49, stackpage=c09a5000)

Stack: 

c09a5e00:                   c00454ec c004550c  60000013 ffffffff fa050000 c0fff630 

c09a5e20: c0972400 c088ba80 c088ba84 ffffffff  c025b120 00492058 c088b980 c09a5eec 

c09a5e40: 00000ec5 c07cdee4 00000001 000000b9  c0047fc4 c09a5eec 00492058 00000000 

c09a5e60: c088b980 c00484fc 00000002 c025b780  00000100 c088b980 00000db9 00000ec5 

c09a5e80: c09a5eec 00000008 00000001 c0048854  00000400 00000ec5 c09a5eec 00000010 

c09a5ea0: c07a33c0 00000400 c078a800 c088b980  021b6a84 00000884 003b1400 c004682c 

c09a5ec0: 021b67a0 c09a5eec 00000000 00000000  c0fff600 000002e4 00000000 00000ec5 

c09a5ee0: c09a5fb0 c0b93d80 c07a33c0 ffffffe4  c09a5f60 00002000 00002000 02170838 

c09a5f00: 00000010 c00853b0 00000000 c09a5f10  00000000 00000000 c037e2bc c09a5f60 

c09a5f20: c09a5f2c c005f21c c09a5f2c 00000000  00000000 00000000 00000000 00000000 

c09a5f40: 021b67a0 ffffffea 00000000 c0b93d20  c09a5ff4 c005f340 021b7308 00001498 

c09a5f60: 00000000 00000000 c09a5f58 00000001  c0b93d80 c0046614 ffffffea 00000b68 

c09a5f80: 021b67a0 02170838 00000010 c0030110  021b67a0 00000010 00000000 c000a108 

c09a5fa0: 000000b5 c09a4000 c09a5ff4 c0009f60  003b111c 00000000 0000002d 021b67a0 

c09a5fc0: 00000b68 003b111c 00000000 0000002d  021b67a0 00000b68 003b111c 02170838 

c09a5fe0: 00000010 be1ffcac be1ffcb0 be1ffc8c  02015a20 020e1724 20000010 0000002d 

Backtrace: no frame pointer

Code: 1a000004 e28630dc (e7932107) e3520000 13a00000 

/sbin/userinit: line 109:    36 Segmentation fault      /empeg/bin/player

.
No recourse except to yank power, attempting a Hijack reboot just yields another segfault.

This kind of crap is obviously not good for the integrity of my disks and my Empeg in general. Tonight was the first time I had gotten pretty far into the sync (about an hour or so in) and I got this. I'm at the end of my rope here... I know I'm supposed to do small syncs but I have a big folder and I don't want to have to do one or two artists at a time...

Back in the 1.03 days I did several syncs of much larger proportion with no problems... Can someone (probably from Empeg) tell me what has happened here and what I can do to avoid it? I have ReserveCache set to 32, is it possible that's causing problems?

Sigh. I wonder if I am going to have severe corruption of my music partition now...

Looks to me like you've run out of virtual memory. This is a Bad Thing.

I seem to remember that there's a swap partition or file lying around somewhere that gets enabled at certain times. Is synchronization one of those times? If not, maybe you could try enabling it before you start your sync and see if it solves your problem.

A little bit of looking reminds me that the swapfile is /swapfile, which, if memory serves, is actually a link to a partition. Before you start your sync, try a ``swapon /swapfile'' and see if that helps. Afterwards, you should probably ``swapoff /swapfile'' to turn it back off. (Then again, it might be turned on during sync anyway, in which case, this is all useless.)

It seems to me that if this is the problem, that it needs to be fixed. I can't come up with a good reason that it would need ridiculous amounts of memory to do this sort of operation. Maybe a transient memory leak?

A little bit of looking reminds me that the swapfile is /swapfile, which, if memory serves, is actually a link to a partition. Before you start your sync, try a ``swapon /swapfile'' and see if that helps. Afterwards, you should probably ``swapoff /swapfile'' to turn it back off. (Then again, it might be turned on during sync anyway, in which case, this is all useless.)

The player software itself enables swap during synchronisation and disables it afterwards.

I can't come up with a good reason that it would need ridiculous amounts of memory to do this sort of operation.

What about

/sbin/userinit: line 109: 36 Segmentation fault /empeg/bin/player

Does it still fail after a clean upgrade (i.e. with our init)? Are you running other stuff besides the player?

Peter

I would have figured that the segfault was probably due to trying to use memory that it failed to allocate due to the VM fault.

I'm pretty sure Tony has a good bit of third-party software running. Maybe he could create an additional swapfile in a music partition and also add that to swap before doing the sync to see if that solves the problem.

Thanks for the responses, guys. Here's the deal... I do have a couple things running, mainly just my alarm clock, which has a pretty small memory footprint, empegVNC, the tiny telnetd that was posted here a while ago... I think that's it... I do have the swap partition enabled, and I do remember it being automatically used during a sync...

I just don't see what the problem could be with memory if it's enabling a big swap partition... But I could try to go back to the normal init and try again. I'm hesitant to do a clean upgrade because it takes a long time to get my Empeg back to my "hacker friendly" setup with all the debian tools, user apps, etc. But in the meantime I'll switch to the stock init and see what's up.

Does that seg fault specifically say I'm running out of memory? I'm no good at reading stack traces.

Are you running Hijack? I have had similar problems with hijack in place. Try installing the default kernel (from the hijack sourceforge site) and see if you have the same problem.

I'm hesitant to do a clean upgrade because it takes a long time to get my Empeg back to my "hacker friendly" setup with all the debian tools, user apps, etc. But in the meantime I'll switch to the stock init and see what's up.

If you have the developer version running (which I assume you have, because of the various tools you installed), and with hijack installed, you have two (or even more) options for backing up your installation:

1. log in to you empeg and do a dd if=/dev/hda5 of=/drive0/program-partition.img bs=4096 (jeesh, I always forget wether the program partition is hda5 or hda4). If you want to go back to your installtion, just reverse the if= and of= options.
2. With hijack, you can also log in using ftp (or even http) and download /dev/hda5 to you PC. To go back to your installation, just re-upload it to the empeg after installing hijack again.
3. With the upgradesplitter/upgradebuilder package that is available on my homepage (www.incase.de), you can also create a .upgrade file from the image of hda5 you created before, but you need gzip also to do so. First: gzip the image, then create an appropriate control file (look at the documentation in the upgradesplitter/builder package for details, you only need the pumphda5 related line, but you should also set the comments and information tags) and finally run upgradebuilder on that control file.
   

I hope to find some time during the following weekend to mess with my MkII for a while. If all goes well, a custom .upgrade file will become available that can be installed over the default empeg .upgrades of the 2.0beta series. That custom file will include hijack, boot from hda2 and include empegVNC, telnetd and some other software I think would be useful.

cu,
sven

Does that seg fault specifically say I'm running out of memory?

No. But the Linux 2.2 kernel can get very crashy if memory is very tight and lots of pages are being dirtied. Asking you to try it after a clean install was just a way of eliminating memory issues from our enquiries.

If you don't want to apply a whole upgrade, just revert init and the kernel to the stock Empeg versions (so nothing but the player is running) and then try it.

Peter

Are you running Hijack? I have had similar problems with hijack in place. Try installing the default kernel (from the hijack sourceforge site) and see if you have the same problem.

Rrrreaaaaallly.... I never heard of Hijack causing sync issues... Wow. Wonder what that's all about... And why that would cause the player app (rather than the kernel) to seg fault?

I will try it without Hijack and with stock init tonight and report progress.. But I would be a little sad if using Hijack and having error-free syncs were mutually exclusive.

With hijack, you can also log in using ftp (or even http) and download /dev/hda5 to you PC. To go back to your installation, just re-upload it to the empeg after installing hijack again.

Yah that's what I was going to do, but it still takes a lot of time. The idea of booting from a second root partition living on hda2 is a really neat idea and I hope that evolves into an "Empeg Standard Development Environment" with all the cool debian tools and stuff already pre-loaded into the image file. As long as we can agree on what to put in there.

Unable to handle kernel paging request

Does that seg fault specifically say I'm running out of memory?

No.

Really? I'm not a kernel or VM expert by any means, but the fact that the kernel failed to handle a paging request screams ``out of memory'' to me.

Rrrreaaaaallly.... I never heard of Hijack causing sync issues...

Well, I have, specifically on my box. I get errors during sync really similar to what you get, except it's ALWAYS during the deleting database step. If I shell into the player and rm -f /drive*/var/database the same thing happens. Of course, when this happens, I can't sync again without a long fsck. Of course then it just happens again...

One thing I've noticed, though, is that flashing the stock kernel makes it work every time. So, whether there's a bug in Hijack or having Hijack just makes memory tight in such a way that my player chokes, I don't know. I'm not running anything besides Hijack, though.

This most recent time that it happened, instead of flashing the stock kernel I set ReserveCache=64 and synced again. It worked that time. I don't know if that's just coincidence or not. If I really need to reserve cache, I hope there's a saner value than 64 that works for me.

Greetings!

I use a reservecache of 32, and I still have the problems. What size are your hard drives? Maybe there is an issue with larger disks. I run 2.0b11 + hijack 253, nothing else. I also see a problem when I upload any file (ftp or through hyperterm), or do any kind of drive work (copies, etc).

One thing I did notice, I think it might only be on the Mark2a. When I had the Mark2, I did not see this problem with sync or shell. It may have just been coincidence, but perhaps it relates to the memory handling since the Mark2 was only 12MB and the Mark2a is 16MB...

I made a few posts on the matter a while back. I am hoping that, when my button kit comes in, I will send the entire empeg off to Mark for installation so he can see the problem first hand. Assuming he is not enjoying himself on a rock somewhere...

Additional:

Well, I tried setting the reservecache to 64, rebooted and just tried to ftp a file up. Crunch! The commands I used in ftp were "bin", "hash", "quote site rw", "put xsl.tar" - Charcoalgrey's XML interface.

It locked up the ftp window, but I had a console session open, and got the following dump. I am now going to spend 90 minutes (45 per drive) fscking the filesystems. That's the only problem with big drives - they take a very long time to check...

Unable to handle kernel paging request at virtual address c10017a8
memmap = C0004000, pgd = c0004000
*pgd = 00000000, *pmd = 00000000
Internal error: Oops: 2
CPU: 0
pc : [<c004550c>] lr : [<c00454ec>]
sp : c0883dc0 ip : 00000001 fp : 00000000
r10: 02e672d3 r9 : c0fb9e60 r8 : c0883dd8
r7 : 00001733 r6 : c0ffba00 r5 : 000012d2 r4 : c0883dd8
r3 : c0ffbadc r2 : 80000000 r1 : 00002000 r0 : 000012d2
Flags: nZCv IRQs on FIQs on Mode SVC_32 Segment kernel
Control: C000517D Table: C000517D DAC: 0000001D
Process kftpd (pid: 24, stackpage=c0883000)
Stack:
c0883da0: c00454ec c004550c 60000013 ffffffff
c0883dc0: c0068844 c0ffba30 c0ff7000 c0fa0a90 c0fa0a94 c01f2540 c0fba1a0 02e672d3
c0883de0: c0fa0990 c0883e94 02e672d3 c0fa0a4c 00000008 00000008 c0047fc4 c0883e94
c0883e00: 00000000 c0fa0990 00000008 c00482c0 00000002 00000001 00000100 c0fa0990
c0883e20: 00000008 00000008 c0883e94 00000008 00000001 c004871c c0883e94 00000b50
c0883e40: c01f7db0 00000010 c0fa09dc 00000400 ffffffea c0fa0990 c0d94000 00001000
c0883e60: 00002000 c004682c c01f7d44 c0883e94 00000000 00000000 c0ffba00 00000000
c0883e80: 00000000 00000008 c07c7194 c07c7180 c0882000 ffffffe4 c01f7d00 00000000
c0883ea0: c0883f20 00001000 00001000 00001000 c0e2119c c00853b0 00000000 c0883ec0
c0883ec0: 00000000 00000000 c0e2119c c0883f20 c0883edc c005f21c c0883edc 00000000
c0883ee0: 00000000 00000000 00000000 00000000 00001000 00000000 c0883f20 00000000
c0883f00: c0883f18 c00ceee0 00000009 00000000 00000001 c0d94000 c0fa09dc c07c7180
c0883f20: ffffffea 00000000 00001000 00000000 c0d94000 c002fa44 00000000 00001000
c0883f40: 00001000 00000000 c088e000 c0883f5c c088e90c c088e70c c00d1c70 00000000
c0883f60: 00001000 c0d94000 00000001 00000000 00000304 00004149 000181a4 00000000
c0883f80: 00000000 00000000 00001000 00000000 3cb3d935 00000000 3cb3d935 00000000
c0883fa0: 3cb3d935 00000000 00000000 00000000 c088e50c 00000001 c088e000 00000000
c0883fc0: c00d2d04 c01c90d0 c088e000 c01341b0 c0102728 c088e000 00000000 c01c90d0
c0883fe0: 00000000 c01341b0 c0102728 c00d3500 c00d3598 00000001 c00d3500 c000b464
Backtrace: no frame pointer
Code: 1a000004 e28630dc (e7932107) e3520000 13a00000

Paul,

I've got a 20 gig and a 12 gig drive in my box. It's almost full, so the database file it has to delete is pretty large, but I'm sure it's nowhere near as large as yours.

I was guessing size related, too, but at 32 gig, mine's certainly not even on the big end of the scale in my book.

My unit is Mark 2A also FWIW. If it is indeed size related, whatever might fix it for you would surely benefit all of us, since you're definitely on the bleeding edge sizewise.

For now, I made some ;@WORK entries in my config.ini to set reservecache to 100 and turn off the servers. I figure that when it breaks I can just yank the power, switch Hijack to Work mode and sync again. Later, when I have patience to endure the fsck, I'll figure out how to reliably break it, and what settings can reliably fix it.

The idea of setting Hijack into Work mode to make my syncs successful reminds me of when friends and I at work would try to debug really difficult problems. After someone had spent hours trying to fix it, someone else would invariably ask if they had considered setting work to true.

Greetings!

Hm... I may have to try the alternate location settings. I still have to find out what values will work, though. The 64 was not enough for me, and since I see the lockup in shell mode without the player running... Still investigating...

I just wanted to put some closure on this problem, as I gave the full sync another try last night with a stock kernel and init. Indeed, the 15 gigabyte sync went off flawlessly. So the only question remaining is if it was the fault of not having enough memory because of user apps, or if it was something with the Hijack kernel. I don't know how to further narrow it down without risking corrupting my music partitions some more... But I guess my gameplan now is whenever I have a huge sync to do (not very often, of course) I'll just back out to a stock kernel, change the init, do the sync, and switch back. A little painful, but it's a workaround.

I'll wait until Mark Lord appears around here again to begin asking questions about what might be actually happening here.

Greetings!

In my case, it is not the size of the sync at all. Even just a simple sync for a configuration change is enough to lock me up. Still, I have pretty much all of the contents up there as I like it so far, so I don't make that many changes. I will continue looking into it, and will investigate further as time and disk allows...

A friend had exactly the same problem whilst running Hijack v253. Not running Hijack fixed the problem.

I'm getting the oops when it gets to the deleting Database line with both v251 and v258, no Hijack, it syncs fine.

Ok, I'm a packrat...I found a copy of v236 on my HD, and flashed it on the player....It sync's fine...I wish I had a few more versions to see which version is the one which broke things.

[edit]

I've found where mlord hid the old versions v243 works, v244 oopses when it tries to delete the database.

[edit]
Oh, HD size 48G, with 17G free currently.

Kernel tracebacks are 100% USELESS unless you also provide the exact *unmodified* Hijack version number, with which they can be very useful..

Cheers

I'll need the Hijack version number to make any sense of the kernel tracebacks.

This doesn't look like an "out of memory" error, but rather some kernel confusion in the ext2 filesystem code.

-ml