Mmm.. now that I've traced through it some, it appears that the player IS doing the RIGHT THING(tm), at least some of the time. I need to play with it more to make sure.

At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.

I'll check that now. Meanwhile, I have a hacked hijack version that automatically ensures that new uploads end up in the right places. Not released yet.

Cheers

Yeah, it's fine.

My worries were for naught. All is well.

But I may still release the changes I made to Hijack, which ensure that the /fids/ directory stays sifted after the initial run.. But then again, a userspace tool can still do this equally well, so why bloat Hijack?

Cheers

I agree that there is no point in bloating HiJack. Could this user ap be executed via ethernet or ftp rather than serial?

Bloat hijack because it's one less thing we'll have to install and keep track of. And it'll be easier to run. Right? Wrong?

Think of this as technical prose, and not really a good solution.

telnet empeg
rw
mv /empeg/bin/player /empeg/bin/player.disabled
cat > /empeg/bin/player << __EOF__
#!/bin/sh
echo "init will helpfully reboot if it can't run the player. Bad init. No biscuit."
return 0
__EOF__
chmod +x /empeg/bin/player
ps auxww|grep player|grep -v grep|awk '{print $2}'|xargs kill
sleep 3
mv /empeg/bin/player.disabled /empeg/bin/player
fidsift
kill -1 1

At database rebuild time, it correctly looks for (eg.) /drive0/fids/100 before it looks for the same file in a subdirectory (eg.) /drive0/fids/_00000/100

So, as long as it remembers where it found each file for later (part of the database?), then it should be fine as is.

It doesn't remember; it re-does the search every time.

Peter

Yeah, but at database rebuild time, it does a directory-recursive search for FID files. At FID-lookup time, it has a specific strategy -- which might be the other way round, I don't recall.

OK Mark I just tried it on a 9436 song empeg - about 50Gb.

My force database rebuild consisted of changing the name of one playlist.

pre sift sync : 4:53
initial sift time was 18:27
post sift sync : 3:20

So not 100% like Brad but still faster. It obviously helps players with more files.

However, there was one problem! On my /drive1 it created a dodgy directory. This is a section of the inital sift...

Sifting /drive1/fids..

mkdir: created directory `_00000'

mkdir: created directory `_0'

mkdir: created directory `_00001'

mkdir: created directory `_00010'

mkdir: created directory `_1'

mkdir: created directory `_00011'

mkdir: created directory `_00012'

mkdir: created directory `_00013'

mkdir: created directory `_00014'

mkdir: created directory `_00015'

hijack: found new-style fids subdirectories

mkdir: created directory `_00016'

mkdir: created directory `_00017'

mkdir: created directory `_00018'

In the _0 directory is now a file called 10 which is the info type file. In the _1 directory is now a file 11 which is the matching mp3 file for the info file in the _0 directory.

According to a CSV export of my database (done after this song was upload I'm 100% certain) the fid of this track should have been 7172 decimal i.e. 0x1C04. So I have no idea what has happened and where it got possible 110 and 111 as the fids from. I don't understand regexps that well, so I can't see anything obviously wrong with your script.

I tried moving the files back to where they were and called them 1c040 and 1c041. I reran the sift program but this time it worked fine and put the files in the correct directory. So I have no idea what went wrong. I haven't listened to the particular song in awhile so it may not have been stored properly, although I didn't receive any sort of database errors via emplode.

Anyway thanks for doing it. It still sped up my rebuilds.

Yeah, I think it is the other way around. Not a big deal, that, though it can be the source of some confusion. Database rebuilds look for non-subdir'd files first, then for the subdirectory versions. At playback time, the player looks into the subdirectories first, and then for the flat file versions. I think.

Rather than theoretically "fixing" that someday, I would very much prefer that a future release of the player software merely include code to automatically write new FIDS to the subdirectories when uploading. Unconditionally would be fine, but one could gate it by requiring that at least one subdir already exists or that the player be empty.

I have modified Hijack-v344 to automatically redirect ALL acceseses to the subdirs, player or otherwise, but only when performed through the symlinks in /empeg/fids?/ (not when done via /drive?/fids/, but only if at least one (any) subdir already exists. I didn't bother with the "empty player" case. To be released soon.

Cheers

Mmm.. Shonky, that's rather odd -- and theoretically impossible from the way the script is written. I suppose maybe /bin/bash got low on memory or something. I will modify the script to turn on swap before/after running, to alleviate any such memory pressures.

The upcoming Hijack-v344 kernel includes code to automatically keep a sifted drive sifted after future sync operations. This code also replaces some existing code that was in the http server, so overall it doesn't contribute much bloat at all.

Cheers

everything except the kill -1 1 at the end (which i hoped would make init respawn the player) worked. fidsifted without use of a serial port last night.

Bump. I thought I'd bump this thread for people interested in fidsift now that the new HiJack is coming.

I decided to send fidsift.sh via TeraTerm (Hyperterminal type program) rather than ftp'ing since it would save me the extra step of hooking up an ethernet cable.

I did an "ro" and "rom" before changing to the root directory and using the "Send File" option.

Problem is, once I sent the file, it appears to have executed the file! Is this normal? It "looks" to be doing the right thing, but I didn't have a chance to be sure... I also never ran "chmod 0755 fidsift.sh".

Do I have anything to worry about? Just let it run? Why did it execute?

UPDATE: Looks like everything ran okay... but I see no trace of fidsift.sh being on the player... It opened the file but never saved it.. I've since ftp'd it to the root directory so it can be run in the future.

Could this line be added to config.ini to run fidsift from HiJack?

[hijack]
;@MENUEXEC fidsift /fidsift.sh

You probably sent an ascii dump rather than transferring the file via xmodem or similar, and since a .sh file is just a series of commands to the shell being run as an interactive shell, it just did the right thing.

What you should have done first is "cat > fidsift.sh", then ascii-dumped it. Then pressed Ctrl-D after it was done to tell cat that you're done sending. The problem with this is that the empeg's serial port uses no flow control, so if there's a buffer overrun or similar, it has no way to compensate, and you'll just lose data. xmodem or similar would be a lot better, but I don't think that there's any such stuff on the empeg to use.

Mark, should I only have folders in my fids folder after running this? I see a bunch of loose fids in there after running it from the HiJack menu... Does the player software need to be turned off or does a "rw" need to be sent prior?

Ahhh.. yes, it probably has trouble running if the player s/w is active. Bummer.. but one could write a fancier shell wrapper for it, that does a seek and destroy on the player before running fidsift, and then restarts the player again after.

Bitt? (I'm overloaded here)

Gosh, I don't have an empeg shell available right now, but it'd be something like:

Code:

---

kill `ps ax | grep player | grep -v grep | awk '{print $1;}'`

fidsift.sh

/player/bin/player

---

I'll double check that soon. I'm sure it's wrong right now (I don't remember the correct player path, or know that awk is installed, or, for that matter, that the empeg's ps puts PID in the first column), but should give someone a start. It should also be possible to fit it into one line for MENUEXEC, but I haven't done any research into how much I can throw at it. I think '&&'s work, so you should just be able to combine those three lines with them.

Anyway, I'll get back to you.

I'm not sure that there's a ps command normally available. If not, then it'll have to parse through /proc/nnn/* to find the player threads.

Cheers

ps is there, at least on the developer install. I forgot, though, that it lists each thread as a different process. And awk is not available. Nor is tail. sed is, though. I'll have to figure out a good script for it. Also, you have to kill -2, otherwise it just restarts the player app. I'll get back to it later today.

someone can just compile busybox really quick and then you can have the killall command. i did it some time ago, but i can't seem to find the binary, and i don't have my player w/ me. maybe tommorow.

Mark, I forgot to mention (and was unable to measure) but the last hard drive upgrade that I did went WAY faster than I expected it to. And it was the largest yet - copying 60gb of data to an 80gb drive. Do you think sifted fids could have helped here? I started mid-evening and didn't have to wait for the copy over night per usual.

So even if people start using Jemplode's rebuld on PC option, it appears that fidsift has other benefits in player performance.

That all makes very good sense -- the kernel in the Empeg just does linear searches through the directory when opening a file, and if the directory is very large, it will have to do multiple seeks/reads and longer searches to find each file..

But newer drives are faster nowadays too, which could also be a factor.

Cheers

Eh, you just need to kill the lowest-numbered PID and all's well. No need to try to kill all of them.

true. but the reason i said to use killall was because you don't have to specify the pid#, therefore achieving your goal trying to use awk, sed, etc. of course, if you have busybox, you'd have those utils, so...

btw, here is busybox

Anyone have any luck with this?