I'm debugging an issue on one of Paul's players where occassionally it throws a wobbly and starts doing erratic things. One failure mode is repeatedly switching between Player and Auxillary every 2 seconds. Another is fast-forwarding through tracks. Both modes are as if the player has received a button down code (IR_RIO_SOURCE_PRESSED or IR_RIGHT_BUTTON_PRESSED) without receiving the corresponding button up.

Paul has indicated that this player has suffered from the problem for a long time. In looking at the problem I've discovered the following in /proc/empeg_ir

Quote:
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 76
Malformed sequences: 3
Spurious transitions: 412
Missed interrupts: 94808
Timings buffer hwm: 64

This is after about 20-21 hours of uptime (Stupidly forgot to check that before rebooting). Missed interrupts are occuring at an average rate of about 75 per minute, and I think that this is the root cause of the problem. Note that the missed interrupts are not regular like clockwork, the actual rate does vary - one minute may have 78 misses, the next 60, and the next 85.

My player which doesn't suffer from this problem had the following after 10 minutes;

Quote:
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 0
Malformed sequences: 0
Spurious transitions: 0
Missed interrupts: 0
Timings buffer hwm: 12

Does anyone know what could be causing a missed IR interrupt? I suspect that this is a hardware fault, hopefully a cold solder joint that could be touched up, but I'm not sure where. Any pointers?

Hmmm!!! I never tried debugging that! Very cool!

I know there were some other folks here who had alternating source (aux / player) problems or other random IR signals. This might be related.

Ok progress.

This is related to the Button LED hack. With the LEDs turned off, the only missed interrupts are a few that occur during the portion of the boot cycle that they are lit. As soon as the LEDs are turned on interrupts start going missing. I'm running some more tests to try and work out whether the quantity that go missing is related to the illumination level or not. First indication is that it is not.

Now as far as I can see there is one very obvious interaction that could be at play here. The PWM current flow through the LED circuitry could be at a frequency that interferes with the IR reception.
Another possibility is that the oscillating current drawn by the knob button is causing a fluctuation in the IR receiver's supply voltage and that the IR supply isn't dealing well with it.

Now this doesn't happen on my player that also has button LEDs, so I'm still wondering what the difference is.

Code:

---

Valid sequences:      52

Repeated sequences:   504

Unfulfilled repeats:  5

Malformed sequences:  880

Spurious transitions: 202

Missed interrupts:    14510

Timings buffer hwm:   58

---

That's off my empeg which I use at home. It's been powered for the last day or so and I've not had any problems with IR at all. [edit]I don't have button LEDs at all as well[/edit]

Shucks. I really didn't want to read that!

Anyway, your missed interrupt rate is significantly lower than Pauls - 14000 vs 90000 in a day, so I'm still not going to rule it out. I've seen only a handful of symptoms, not 90000, do if this was the cause then 14000 might not have been enough to make it occur in your player.

Of course, now I have to wonder why your player without button LEDs has this missing interrupt issue when there seems to be a link on Pauls player, and my player doesn't suffer at all.

Could it be the age or the IR receiver growing faulty with time?

Code:

---

Valid sequences:      52
Repeated sequences:   544
Unfulfilled repeats:  5
Malformed sequences:  963
Spurious transitions: 324
Missed interrupts:    24497
Timings buffer hwm:   62

---

18 hours or so later and it's now this so I seem to get about 14000 missed interrupts a day. I've no idea what is causing this however as that particular empeg is pretty much stock apart from the new single Toshiba 60G HD inside.

Ok, during a loopy period, the rate of missed interrupts is nearly 500/minute.

Hmm. It turns out, having just repeated some experiments, that maybe the LEDs really don't have anything to do with it. Even with the LEDs off now I am seeing missed interrupts. I cannot explain why the earlier experiment gave the opposite result. Was that just a fluke?

I've also re-confirmed that my player doesn't ever drop interrupts. The differences between Paul's players and mine are that his have 160GB vs 60GB (non-related, surely?), and that his is running v2 final whereas I'm running v3a7. Both have the same hijack version (v394). Oh, mine has 64MB RAM, his 48MB.

Could it be a software issue? I don't think so. Trevor, what are you running?

v3a7, Hijack v383, 16MB RAM, single Toshiba 60GB HD and one of Patrick's tuner kits. This is all very strange.

I've had a poke around in the kernel source and everywhere which increments that specific counter has "/* Shouldn't happen */" next to it. I don't know if this means that it's not ever possible in a valid IR sequence or if it's not ever possible in the code. It's detected a couple valid sequences even though I've not used the empeg remote in the last day.... Hmm...

Soitenly. I have a player (Hermit) at the office, so I will just let it random play for a while.

By the way, initial state before playing...

empeg:/empeg/bin# cat /proc/empeg_ir
Valid sequences: 0
Repeated sequences: 0
Unfulfilled repeats: 0
Malformed sequences: 0
Spurious transitions: 0
Missed interrupts: 0
Timings buffer hwm: 31

Edit: Two hours later, all of the values are exactly the same. But I have not used the IR or buttons, really...

Paul,

The good news is that your player is not currently suffering from this problem.
The bad news is that I want my display board back.

The problem definitely moves with the display board. I suspect that I'm not going to get any answers soon, and that I'm going to end up shipping Traveller back as-is, with the advice that swapping the display with one of your spares is a good idea.

No worries. If it is localized to the display board, that really limits the amount of items it could be. I suspect that it is strongly related to the IR receiver itself.

I can always transplant the display board into a lesser used player or spare (Archive), along with a note describing the problem.

Thanks for tracking and identifying this! At least I know I wasn't going crazy!!!

WOW!!! Excellent!!! Great detective work!

This might be a good FAQ topic. I know I was not the only person having this trouble. But it was so intermittent (most of my listening in the car is short bursts with lots of button presses, so the problem did not have much of a chance to manifest itself), I did not try any debugging.

So refresh my memory. What exactly were the symptoms leading up to the desire to go hunting in proc/empeg_ir?

Occassional random loopiness occurring every few hours, as if the player has received a button code. Possible incarnations include (in order of observed frequency);

Rapid switching between Player and Aux as if someone is stabbing the 'Source' button every 2 seconds.
Continuous Fast Forwarding through tracks as if the right button is being held down.
Player shutting itself down as though the top button were held down.

In all cases, a valid button press, knob twiddle or IR code breaks the cycle and restores proper behaviour. There may be other similar symptoms that I didn't see.

If this problem is suspected then the test is to check /proc/empeg_ir, both immediately after boot and then after the player has run for a couple of hours.This can be done;

from a shell prompt by typing "cat /proc/empeg_ir". or
through hijack's web interface by visiting "http://empegs_ip_address/proc/empeg_ir".

Within that output we are interested in the "Missed Interrupts" figure. It should be considered normal if a few (<20) are seen at bootup. After a couple of hours, numbers in the thousands are cause for concern. A player reporting ~9000 is known to suffer from the loopiness. A player reporting ~3000 has not been witnessed suffering, although it's possible that it does, only less frequently. A good player has a negligible or zero increase from the value reported after boot.

Done.

We've got an MK2 player here which doesn't take inputs from the remote or via buttons or encoder except on very rare occasion when a lot of the buttons are mashed at once and repeatedly, an occasional command will squeek through, usually repeated over and over until the unit is shutdown. We don't see huge numbers of missed interrupts although there are some. All the connections to the PIC from the pushbuttons and encoder test good as does the cable connection to the board. We've tried resoldering the IR receiver but that didn't change anything.

Anyone else seen this and know how to correct it?
Stu

I'm sure you've done this, but have you swapped the display board to see if it's a problem on that board or not?

Oh yes, sorry forgot to mention that. Yes swapping displays cures the problem.

Thanks though,
Stu

I assume you swapped the display board with a known good one at this point, and identified that the problem is on the display board. Check the connections around the buttons - maybe something is broken there, and is flooding the input.

Yep, we had checked all that out too.

Thanks,
Stu

Bad PIC? Is the button input controlled by a PIC, and is that PIC on the display board?

Could be a bad PIC except that remote commands don't work either and I don't believe these interface with the PIC. This PIC (I believe) converts button presses into IR data, but it is sent over a different line to the main board than the IR receiver. Yes this PIC is on the display board.

Thanks,
Stu

From Patrick's post here button presses are sent down the same line as remote commands, if I understood it correctly.

Quote:
Button presses are actually encoded as IR signals, and sent down the CIR RX line

Stig

This is probably the button PIC resetting continuously. Probably means either the PIC or the reset circuit has gone bad. Seen it a few times, fixing one now.

Yeah I see now the IR receiver and buttons/encoder go to the same PIC. We had checked the reset pin previously, and it is just a resistor to 5 volts I believe.

Update:
We found the problem. It was a bad solder joint at pin 2 of the PIC near the buttons. Although it showed continuity to the IRRX line, the signal was not reaching the PIN reliably. After a quick touch up of the solder joint, control seems 100% again. Apparently the slight pressure of the probe was enough to establish continuity were none was without the probe. Tony, it would be great if you could add this to the FAQ.

Thanks,
Stu