I have just released mp3tofid version 3.00 , to be found here as usual.

There are three major new features

• support for (and defaulting to) the new player directory structure, ie. fids named /driveX/fids/_YYYYY/ZZZ. Use "-o" to revert to the old structure.
• intermediate database for storing inode to fid relationship, resulting in a much smaller player database
• a Windows port (actually a cygwin port)
  

Until now, it seemed mp3tofid only worked for me. Previous versions seemed to produce much too large databases for the player to handle. The database size would depend on how inode numbers were distributed on the filesystem containing the source MP3's. In this version, this should no longer be of influence.

The intermediate database is stored as .../drive0/var/mp3tofid. Don't lose this database. A new one will be created if it is lost, but all tunes will get new fid numbers resulting in a total re-upload if you run rsync again.

As a result of its location, the mp3tofid database will be copied to the player. It serves no purpose on the player, but it might be handy to have it stored as a backup there.

Pim

So far, it looks great. My 600 meg database has dropped to 229k. I haven't synced it all to my empeg yet, I'll try that tomorrow. Thanks again.

i'm getting this error when using rsync
skipping non-regular file "drive0/fids/_00000/8f0"

any help?

It's probably a dangling symlink. Have you checked with ls -l?

You can also use symlinks (written by Mark Lord) to check for bad symlinks.

Another possibility is that the rsync server does not have permission to read the file the symlink is pointing to.
The rsync server usually runs as nobody on Unix or SYSTEM on Windows, so the target should be world-readable.

Pim

its because i ran mp3tofid w/ drive letters (e:/mp3 e:/empegfids)instead of using /cygdrive/e/. its been syncing since last night.

another question.... i'm looking at the fids directory.... theres a blank files that aren't connected to files everywhere. is that by design?

i ran mp3tofid w/ drive letters (e:/mp3 e:/empegfids)instead of using /cygdrive/e/.

Ok, now I understand you're using cygwin. I was not aware windows-type paths worked at all in cygwin ...

theres a blank files that aren't connected to files everywhere

You mean zero-sized files? There should not be any.

You'll see three types of files:
- tag files, ending with "1". These are small (unix) text files.
- playlists, ending with zero. These are small binary files.
- symlinks, ending with zero. They point to you mp3's.

symlinks can only be seen as such by cygwin apps. Native windows apps see them as shortcuts, but they are twisted. Dont try to edit the shortcut properties. True shortcuts do not appear as symlinks on cygwin, but as plain files with a .lnk extension.

Pim

ok, i was wondering what the tag files were. now i know. but this poses another problem. is there anyway for mp3tofid to re-create the tag files if i modify them on the hdd? this was the whole point of me getting this to work (aside from the knowledge =).

ok, i was thinking this over today. if mp3tofid can check the modified date versus the last time mp3tofid was run, if its a match, delete the tag file and regenerate.

also, can there be a switch to check for id3v2 tags only? i see that mp3tofid is scanning my huge compilation cds wholy, and i was wondering why?

if mp3tofid can check the modified date versus the last time mp3tofid was run, if its a match, delete the tag file and regenerate

This is basically the algorithm:
1. read all existing fids into memory
2. for each fid, do numerous tests to see whether it still matches the source tune, otherwise delete from memory
3. scan the source mp3 tree, skipping all tunes that have valid fids in memory
4. completely empty the fids directories
5. regenerate all fids and databases from memory
6. set modification date of tune fids and tune tag fids to their source file's modification date (this speeds up rsync a lot)

also, can there be a switch to check for id3v2 tags only?

It is libid3tag's behaviour to look for id3v2 tags and drop back to id3v1 if no id3v2 tags at all are found.

i see that mp3tofid is scanning my huge compilation cds wholy, and i was wondering why?

I could not find a reliable way to calculate duration and bitrate from an mp3 using libmad, other than by decoding each frame header, essentially reading the whole file. It's only decoding the headers of the frame, not the frame itself, so this is an i/o bound operation. On my system, it took 110 minutes to scan 72 GB of MP3's. (It took 30 hours to upload using rsync in one single run).

Still, it only has to be done once; on next runs, only new or changed tunes get scanned, and only new or changed fids are uploaded by rsync.

If you have a huge amount of tunes on your player, adding a single album using mp3tofid+rsync might take less time than it takes emplode+player just to rebuild the database.

But speed was not really the most important issue for me to write mp3tofid. I just did not want to keep track of all changes and apply these changes to my player manually.

I keep changing things to my mp3 collection, like
- changing id3 tags
- re-ripping badly ripped cd's
- re-encoding using better encoders or encoding options
- renaming tunes and directories
- deleting tunes and directories.
- adding tunes, of course
Whatever I do, mp3tofid and rsync will see the changes and apply them to the player automatically.

If you are going to upload a huge amount of tunes a first time using rsync, you may consider to mke2fs your fid filesystems before doing so (don't forget to save your config.ini file). Uploading to a freshly created, empty fs is much faster than overwriting old stuff. You will also get rid ot the huge fids directories that stay huge otherwise, even if you delete their contents.

Pim

ok, let me clarify my predicament. i noticed yesterday that one of my tags aren't capitalized right (Dj instead of DJ). i guess the tests doesn't check for capitalization changes.

but otherwise, i'm enjoying the mp3tofid/rsync otherwise. can't wait til ignore as child and "other" file types gets implemented. thanks for developing it.

i noticed yesterday that one of my tags aren't capitalized right (Dj instead of DJ). i guess the tests doesn't check for capitalization changes.

Well, the windows filesystem is case insensitive, even under cygwin. So if you only change capitalization, mp3tofid will still be able to open the file, and conclude it did not change. But if you edit the id3 tag inside the file as well, the modification date ought to change, and mp3tofid would notice. Perhaps your id3 tag editor resets the modification date after its change?

can't wait til ignore as child

I know what you mean. The other day, my player randomly played a 0db sine wave off my test cd ...

and "other" file types gets implemented

Which other types in particular? WMA is out, because of its closed nature. OGG and FLAC should be possible, I guess.

Pim

let me try this again. yeah, your right. mp3tagstudio doesnt modify the date. it works, just user error.

as for unsupported files, i need to sync up my wav files so i can do my test tones also. i just dont believe in mp3 test tones..

and how about this for a feature.... the ability to put a jemplode only soup file in a directory (probably in var), and let mp3tofid interpret it and convert to a playlist.

argh. check out what i did. i took off the "keep original date/time" toggle, and applied a tag renaming scheme to my whole collection. mp3tofid catches this, and rescans. there are many mp3s that stay the same, but just had the modified date changed.

when i try to rsync, it starts to do the whole collection again. i guess it only checks if theres a change in the modified date. i know this is not mp3tofid's fault, but isnt there a switch to make rsync make sure that nothing was changed?

isnt there a switch to make rsync make sure that nothing was changed?

You shouldn't need to worry; rsync will detect the files did not change; both ends are exchanging checksums. But both ends will need to read the whole file, so it will be slower. As a result, the fids on the player will get the modification time of the source fids, without actually transferring their contents.

Pim

as for unsupported files, i need to sync up my wav files

As wav files do not contain tags, I would have to add code that takes the title from the filename. This would be useful for mp3 too, as this is what emplode does now if a title tag is not found. I'll put that on my todo list.

the ability to put a jemplode only soup file in a directory

Uhm, now's the time to shamefully admit I have never tried to use jempeg ...

Pim

question.. i'm going to start using a preinit script to run rsync on startup, so i need to know if its safe to run it while the player app is still up? and if it eventually gives up if it doesnt find a network connection (i.e. in the car).

is there any sure way to know that i even have a network connection? somewhere in /proc maybe? so i can make the script conditional.

also, even though the database and playlist is functional, everytime i run emplode after rsync is done, it has to add/change something (litte red cross). i was thinking of doing a file compare, so you can implement what happens to the database/playlist generation code. or you can do it yourself.

ok, another feature request. i have different size hard drives... and i currently use the -2 50 switch. i need a celing on the smaller drive so i dont go over capacity. but until the time comes, i want to split up the mp3s evenly.

I just finished rsyncing the new fids and all to my empeg, and it works great now. No problems at all. I love the program, thanks so much for writing it.

is there any sure way to know that i even have a network connection?

If you get your IP from a DHCP server, you can see with ifconfig that you did not get an IP number.Get ifconfig from a Debian image.

somewhere in /proc maybe? so i can make the script conditional.

you can see in /proc/empeg_power whether you're on AC power or on car power.

even though the database and playlist is functional, everytime i run emplode after rsync is done, it has to add/change something

I noticed that too. I think it's because of some timestamp, not because of something in the database. I tested this in an earlier mp3tofid version; the databases created by emplode and by mp3tofid were 100% identical. With current versions they are not, because mp3tofid is leaving some tags out of the database that do not have to be there. This makes the database, and thus the player's memory usage, significantly smaller.

Pim

I have different size hard drives... and i currently use the -2 50 switch. i need a celing on the smaller drive so i dont go over capacity

In order to prevent unnecessary upload, it has to be predictable for any single file on which drive it will end. So I can't dynamically put the the file on the drive with the most diskspace like emplode does.

If it turns out one drive is getting full, then just change the percentage (ie "-2 51" or "-2 49"). You can check the diskspace on your host pc using du -L -s .../empegfids/drive?

I guess this makes it hard to fill both drives to the last byte, but you can't have everything in life ...

Pim

i'm going to start using a preinit script to run rsync on startup, so i need to know if its safe to run it while the player app is still up

I guess it would be safe to run rsync as a daemon while the player app is running, and drives are mounted read-only.

I would guess it is not safe to write to fids and databases while the player app reads them.

Pim

i have found that running rsync while the player is running is incredibly slow even while the player's on standby. i'm guessing the player app is running at a higher cpu priority and all memory is filled with song data.

which leads me to ask... is there anyway aside from hitting q in terminal mode to kill the player? i've tried to just use kill -9 pid but it always restarts.

is there anyway aside from hitting q in terminal mode to kill the player?

I wouldn't know. Perhaps it is best to ask again outside the context of this thread, and get a wider audience. I'd be interested too...

Pim

It's evil, but you can:
-move the binary out of the way
-kill "player"
-init has nothing to respawn
-don't forget to move it back or you will be sad later

You could use kill -STOP <playerpid> which should freeze the player. Run rsync and then kill -9 <playerpid>, and the player will restart and reload the new databases. Although the player is a threaded process, so I'm not sure whether you can just kill -STOP the lowest player pid, or if you have to stop all of them individually.

I guess I spoke too soon.

I just added a new album and re-ran mp3tofid, and it acted as if I had changed all the files - rescanned every file, and generated a new fid for every song. I was suspicious, so I ran rsync with the dry run option, and sure enough, it wanted to delete everything on the player and start over again.

I am sure this is something to do with me using XFS on my server, but I doubt that XFS randomly reassigns inode numbers whenever it feels like it. I am wondering if I am overflowing anything with the high inode numbers (the latest songs I just added all have inodes in the 373 million range).

Can you reproduce this, using a subset of your collection and a test fid dir?

I doubt that XFS randomly reassigns inode numbers whenever it feels like it

It's unlikely, but not impossible. This is easily detected. Just type ls -li in an mp3 dir, reboot and look again. smbfs on Linux turned out to give different results on each boot.

I am wondering if I am overflowing anything with the high inode numbers (the latest songs I just added all have inodes in the 373 million range)

There's nothing that can overflow anymore (at least not sooner than with other tools). A tune is now uniquely identified by a string built by the major and minor number of the device holding the filesystem containing the tune and the inode number of the tune. This string is used as a key to look up the fid number in a gdbm database. using -d dimMsS debug options you can follow all gdbm activity.

In theory, a 64-bit inode number (yours still look like 32-bit) could overflow the sprintf() function, but then still it would come up with a unique string.

So I can think of a number of explanations for this behaviour:
- your filesystem does not have persistent inode numbers
- you somehow lost the mp3tofid database
- you used the -n switch to force generating a new mp3tofid database
- the major/minor number of the filesystem changed caused by fdisk or lvm operations
- a weird bug

If you did lose the mp3tofid database, by accident or by using the -n switch, try restoring it from the player. This is why it is backed up there. Restore it, delete the fids on your host PC and run mp3tofid again. mp3tofid will have to scan everything again, but your fid numbers will be back to what they were.

Pim

I'll do some testing. I didn't delete the database or use -n though. And I am using EVMS, but one of it's features is persistent major/minors. I'll let you know what I find out.

A lightbulb just went off in my head. Permit me to place my foot in my mouth.

I still have to test, but I am sure it's the major/minor issue. What I said is true, one of the features of an EVMS native volume is persistent major/minors. However, the volume that my mp3s are on is not a native, but a compatibility volume (dynamic major/minors) because I haven't had time to convert it yet. And I am pretty sure I did something to cause it's minor number to shift between mp3tofid runs.

So that must be the cause. Can I suggest to add a very visible section to the README about this, and I'd mention LVM volumes and EVMS compatibility volumes specifically. EVMS native volumes won't have this problem.

Thanks..

here's what i've discovered about the windows version. if i modify a file using winamp (or anything else for that manner), mp3tofid immediately recognizes it as a new file (i see 1 empty fids in the summary screen), and deletes the old fid. not sure why.

but the weird thing is... rsync says it only wrote < 10000 bytes, but took 10 seconds to complete after the initial scan. i dont understand whats happening in rsync i guess.