I just had my mk2a upgraded to 32 MB. I loaded hijack v391 before upgrading.
After turning it on, it passed the memory test, but unfortunately the kernel
panic's soon afterwards. It always panics or oopses at the same spot.

I tried v390, v391 and v392 without luck. Hugo's bootloader did not have
effect either. Reverting back to v380, which I used before makes the
problems (and the extra memory) go away.

Here's a dump of a v392 oops using Hugo's e000.rom.

So, is this a hardware or a software problem?
Maybe we need a more thorough memory test?

Thanks,
Pim

empeg-car bootstrap v1.02 20001106 (hugo@empeg.com)

If there is anyone present who wants to upgrade the flash, let them speak now,

or forever hold their peace...it seems not. Let fly the Penguins of Linux!

 

e000 v1.04

Copying kernel...

Calling linux kernel...

Uncompressing Linux..................................... done, booting the kernel.

Linux version 2.2.17-rmk5-np17-empeg52-hijack-v392 (root@localhost.localdomain) (gcc version 2.95.3 20010315 (release)) #2 Mon May 24 15:31:28 PDT 2004

Processor: Intel StrongARM-1100 revision 11

Checking for extra DRAM:

c1000000: passed.

c1100000: passed.

c1200000: passed.

c1300000: passed.

c1400000: passed.

c1500000: passed.

c1600000: passed.

c1700000: passed.

c1800000: passed.

c1900000: passed.

c1a00000: passed.

c1b00000: passed.

c1c00000: passed.

c1d00000: passed.

c1e00000: passed.

c1f00000: passed.

c2000000: wrote ffffffff, read 00000000

NetWinder Floating Point Emulator V0.94.1 (c) 1998 Corel Computer Corp.

empeg-car player (hardware revision 9, serial number 40104351)

Command line: mem=16m

Calibrating delay loop... 207.67 BogoMIPS

Memory: 31240k/32M available (980k code, 20k reserved, 524k data, 4k init)

Dentry hash table entries: 4096 (order 3, 32k)

Buffer cache hash table entries: 32768 (order 5, 128k)

Page cache hash table entries: 8192 (order 3, 32k)

POSIX conformance testing by UNIFIX

Linux NET4.0 for Linux 2.2

Based upon Swansea University Computer Society NET3.039

NET4: Linux TCP/IP 1.0 for NET4.0

IP Protocols: ICMP, UDP, TCP

TCP: Hash tables configured (ehash 32768 bhash 32768)

IrDA (tm) Protocols for Linux-2.2 (Dag Brattli)

Starting kswapd v 1.5

SA1100 serial driver version 4.27 with no serial options enabled

ttyS00 at 0xf8010000 (irq = 15) is a SA1100 UART

ttyS01 at 0xf8050000 (irq = 17) is a SA1100 UART

ttyS02 at 0xf8030000 (irq = 16) is a SA1100 UART

Signature is 206f6972 'rio '

Tuner: loopback=0, ID=-1

empeg display initialised.

empeg dsp audio initialised

empeg dsp mixer initialised

empeg dsp initialised

empeg audio-in initialised, CS4231A revision a0

empeg remote control/panel button initialised.

empeg usb initialised, PDIUSBD12 id 1012

empeg state support initialised 0089/88c1 (save to d0004180).

empeg RDS driver initialised

empeg power-pic driver initialised (first boot)

RAM disk driver initialized:  16 RAM disks of 4096K size

empeg single channel IDE

Probing primary interface...

hdb: IC25N080ATMR04-0, ATA DISK drive

hdb: IC25N080ATMR04-0, ATA DISK drive

hda: IC25T060ATCS05-0, ATA DISK drive

hdb: IC25N080ATMR04-0, ATA DISK drive

ide0 at 0x000-0x007,0x038 on irq 6

hda: IC25T060ATCS05-0, 57231MB w/1768kB Cache, CHS=7296/255/63

hdb: IC25N080ATMR04-0, 76319MB w/7884kB Cache, CHS=9729/255/63

empeg-flash driver initialized

smc chip id/revision 0x3349

smc9194.c:v0.12 03/06/96 by Erik Stahlman (erik@vt.edu)

                                                        

SMC9194: SMC91C94(r:9) at 0x4008000 IRQ:7 INTF:TP MEM:6144b MAC 00:02:d7:28:10:ff

Partition check:

 hda: hda1 < hda5 hda6 > hda2 hda3 hda4

 hdb: hdb1 hdb4

RAMDISK: ext2 filesystem found at block 0

RAMDISK: Loading 320 blocks [1 disk] into ram disk... <1>Unable to handle kernel NULL pointer dereference at virtual address 00000020

memmap = C0004000, pgd = c0004000

*pgd = c012f801, *pmd = c012f801, *pte = c000000b, *ppte = c000000a

Internal error: Oops: 0

CPU: 0

pc : [<c002a9a4>]    lr : [<00000020>]

sp : c0003828  ip : c1f98fa0  fp : c0003844

r10: 00000000  r9 : c0103f38  r8 : c0003868

r7 : 00000005  r6 : 00000000  r5 : c0103fd8  r4 : c0181740

r3 : 00000000  r2 : e58bc018  r1 : 00000005  r0 : c0181740

Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  Segment kernel

Control: C000517D  Table: C000517D  DAC: 0000001D

Process swapper (pid: 1, stackpage=c0003000)

Stack:

c0003800:                                                        00000020 c002a9a4

c0003820: 60000093 ffffffff 00000000 c0103fd8  00000000 00000400 c0003860 c0003848

c0003840: c0033f14 c002a940 c1f98f20 00000800  00000000 c0003898 c0003864 c0033fdc

c0003860: c0033ea8 c1f90000 c0002000 00000000  00000000 c1f90000 00000400 00000100

c0003880: c0103f98 c0103f38 00000048 c00038b8  c000389c c003493c c0033f9c 00000400

c00038a0: 00000000 00000001 00000100 c00038cc  c00038bc c003370c c00348d4 00000400

c00038c0: c00038fc c00038d0 c00338d8 c0033708  c00038fc 00001000 00000001 00000000

c00038e0: 00000000 00000400 00000000 00000048  c0003b58 c0003900 c003760c c00337b8

c0003900: 00000001 00000100 00001000 00000000  00000000 00000000 00000000 00000400

c0003920: c0003c68 00000400 c0191800 c0003c54  00093fa7 0800b2a1 ba03a7ff 6a949b67

c0003940: 4100008c 0246dd49 885805ff 8645bf6f  a10398ea 060070d8 8e5c873f 6a11d3de

c0003960: 00401843 40121458 8431ebed af07aa77  9300eedb 1a20329c ba52bed7 ac15f7df

c0003980: 04114800 a050f8b3 2e7567d7 240028d3  8000f24b a0647f7f ae53bdff 5a601fae

c00039a0: 40180e90 cf57fae7 2b15ff7c 008926c8  00123d3a bae7fef1 2b5169d7 041230c4

c00039c0: 084495af 5255fc1b ac4f8fad 100430bd  a0000013 c0003a30 c010d9ac c0003a00

c00039e0: c00039ec c000f580 00000000 0000001a  00000340 00000001 c0003a20 c0003a08

c0003a00: c00d51a8 c00bae30 00000000 00000000  c012c210 c011f244 00000000 c0121d68

c0003a20: c01120f0 c011f244 00000000 c0121d68  c0003a58 c0003a40 c00bb19c c001560c

c0003a40: c011f1d0 00000000 00000000 c0003a68  c0003a5c c00bbce0 c00bb0cc c0003a78

c0003a60: c0003a6c c00cf2e4 c00bbcd8 c0003a90  c0003a7c c0015d54 c00cf2e0 00000001

c0003a80: c0121dcc c0003aac c0003a94 c001e820  c01946e0 00000003 00000060 00000000

c0003aa0: c0003ad8 c0003ab0 c000ab68 c000a9d0  fa050000 c0003b24 00000001 c00e427c

c0003ac0: 20000013 ffffffff c0003c54 c0003b58  c0003adc c0009e60 c000aa6c c01918a4

c0003ae0: d00bd8a4 00000000 0affffec c00f4513  c011273b c0109018 00000002 00000000

c0003b00: 00000002 c011273e c0003b48 c0003b18  c0019830 c001560c 00000000 00000002

c0003b20: 00000040 00000400 00000400 c0003cc0  00000048 00000048 c0191800 c0003cac

c0003b40: 00000000 00000140 c0003c54 c0003f84  c0003b5c c00a1100 c00374a4 00000140

c0003b60: 00000005 00000000 00000000 000001fa  5c2d2f7c 00000000 00000000 c0003d04

c0003b80: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003ba0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003bc0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003be0: 00000000 00000000 00000000 c0003e30  00000000 00000000 00000000 00000000

c0003c00: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003c20: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003c40: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 c0003b74

c0003c60: c0108798 00000003 00012000 00000000  00000000 00000000 00000001 00000000

c0003c80: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003ca0: 00000000 00000000 00000000 00000000  00000000 c0003be4 c010875c 00000001

c0003cc0: 00012400 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003ce0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003d00: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003d20: 00000000 00000000 00000000 01000000  00000000 00000000 00000000 00000000

c0003d40: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003d60: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003d80: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003da0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003dc0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003de0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003e00: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003e20: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003e40: 00000000 00000000 00000000 00000000  00000000 00000000 01fa0000 00000000

c0003e60: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003e80: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003ea0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003ec0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003ee0: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003f00: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003f20: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003f40: 00000000 00000000 00000000 00000000  00000000 00000000 00000000 00000000

c0003f60: 00000002 00000080 c012aea0 c0122540  4401a11b c0008580 c0003f94 c0003f88

c0003f80: c00a12e4 c00a0d48 c0003fb4 c0003f98  c00468ec c00a12d4 00000001 c0100768

c0003fa0: c012188c c0108754 c0003fdc c0003fb8  c0009ac0 c00467b4 00000000 c0009b88

c0003fc0: c012b80c c0100760 c0100764 c0100988  c0003ffc c0003fe0 c0009b98 c00099b4

c0003fe0: c0009b88 c012b80c c0100760 c0100764  c00fffd8 c0004000 c000b6cc c0009b94

Backtrace:

Function entered at [<c002a934>] from [<c0033f14>]

 r7 = 00000400  r6 = 00000000  r5 = C0103FD8  r4 = 00000000

Function entered at [<c0033e9c>] from [<c0033fdc>]

 r6 = 00000000  r5 = 00000800  r4 = C1F98F20

Function entered at [<c0033f90>] from [<c003493c>]

 r10 = 00000048  r9 = C0103F38  r8 = C0103F98  r7 = 00000100

 r6 = 00000400  r5 = C1F90000  r4 = 00000000

Function entered at [<c00348c8>] from [<c003370c>]

 r7 = 00000100  r6 = 00000001  r5 = 00000000  r4 = 00000400

Function entered at [<c00336fc>] from [<c00338d8>]

 r4 = 00000400

Function entered at [<c00337ac>] from [<c003760c>]

 r10 = 00000048  r9 = 00000000  r8 = 00000400  r7 = 00000000

 r6 = 00000000  r5 = 00000001  r4 = 00001000

Function entered at [<c0037498>] from [<c00a1100>]

 r10 = C0003C54  r9 = 00000140  r8 = 00000000  r7 = C0003CAC

 r6 = C0191800  r5 = 00000048  r4 = 00000048

Function entered at [<c00a0d3c>] from [<c00a12e4>]

 r10 = C0008580  r9 = 4401A11B  r8 = C0122540  r7 = C012AEA0

 r6 = 00000080  r5 = 00000002  r4 = 00000000

Function entered at [<c00a12c8>] from [<c00468ec>]

Function entered at [<c00467a8>] from [<c0009ac0>]

 r7 = C0108754  r6 = C012188C  r5 = C0100768  r4 = 00000001

Function entered at [<c00099a8>] from [<c0009b98>]

 r8 = C0100988  r7 = C0100764  r6 = C0100760  r5 = C012B80C

 r4 = C0009B88

Function entered at [<c0009b88>] from [<c000b6cc>]

 r7 = C0100764  r6 = C0100760  r5 = C012B80C  r4 = C0009B88

Function entered at [<c00096c4>] from [<c00098a4>]

Function entered at [<c00096dc>] from [<c0008080>]

 r7 = C0121D60  r6 = C0121CB4  r5 = C012C240  r4 = C012C240

Code: e3530000 1a00001a (e58ec000) e5943008 e063500e

Some extra testing:

Using Hugo's e000.rom, hijack v386 and v387 don't crash, but don't detect
the extra memory either. v388 does detect 32MB, but crashes.

Pim

This part of the bootlog made me wonder:


RAMDISK: ext2 filesystem found at block 0
RAMDISK: Loading 320 blocks [1 disk] into ram disk... <1>Unable to handle kernel NULL pointer dereference at virtual address 00000020


Why does the boot process require a ramdisk? At this point, the harddrives
are already detected, so why not just mount the root filesystem immediately?

And where is the data store that is loaded into the ramdisk?

Pim

And where is the data store that is loaded into the ramdisk?

In flash. I don't remember the address, though.

The initrd (ramdisk) contents are in flash at 0b0000, and are reprogrammed by any s/w upgrade. I'm not sure what the initrd is used for, since the linuxrc executable therein is a binary (rather than the usual simple command script).

But regardless -- your memory is bad. The Hijack DRAM test is not exhaustive, nor failsafe, since achieving those goals would significantly slow down boot times. But odds are very good that the data pins are fine, and the problem you have is with the address lines. Touch all of those up with the iron and try again.

Cheers

Question (knowing I am asking for trouble):

Could a version of the kernel be put together for people installing / testing memory upgrades that would give an exhaustive memory test, just to verify the installation. This was somewhat fortunate, given that it panicked early. However, if a different spot in memory was corrupt and it was not seen until later on, it could be much more difficult to identify. Especially if it did not cause a panic.

Testing DRAM is very complex, and difficult to get right. Each mem write has to be verified to go to one and only one location. It has to check for stuck bits, crossed lines, shorted lines, and failed refresh. And also has to deal with the CPU caches (simply marking the sections as "non-cachable" doesn't seem to be enough..).

Even the code I have there now was difficult to get working, and I'm still not sure if it has successfully defeated the various caching mechanisms completely.

So, in short, I could produce a special memtest kernel if someone else provides the working code. Otherwise, it's a tough nut to crack.

How does the existing memory test (ctrl-T at boot) do it? If it can see the added memory space, a special kernel would not be needed.

probably just a stupid idea (I'm full of them), but can't the 'special' version just loop around writing/reading patterns too all the detected memory forever and just print errors when it doesn't read back what it wrote. If you write the whole 16/32mb before reading it back, won't the cache have been flushed, as there is no way it can cache that much.

Anyway just a thought.

What is our immediate purpose? To test the RAM itself or the installation? Are all the address lines tested with a 0x00,0x55,0xaa,0xff pattern like the data lines are, or do only the high address lines get hit as sparse locations are tested?

I'd argue that the nature of the upgrade merits hitting all of the address lines, but that a full memory test is not necessary for the Mk2/Mk2a (assuming that the chips were handled properly.)

The Mk1 is an interesting case. We know from Hugo that the Mk1 chips have a known high failure rate, and we also know that the upgrade itself is easier and less prone to problems.

Anyway, http://www.esacademy.com/faq/docs/memtest/index.htm has useful strategies for testing data lines, address lines and a full device test. They even have sample code that could be adapeted.

Pin A3 needed some extra solder. Everything's fine now.

Thanks,
Pim