Unoffical empeg BBS

Quick Links: Empeg FAQ | RioCar.Org | Hijack | BigDisk Builder | jEmplode | emphatic
Repairs: Repairs

Topic Options
#249610 - 16/02/2005 22:05 empeg backup utility
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
Hello all... i'm not sure if this has been done already. I dont think i've seen this yet, so i'll propose the idea and wait to see what ya'll think. Maybe i can get some help from some programmers on this board.

I was thinking about creating a backup util for my empeg. my thoughts are these:

1. I've got about 100 gig of mp3s on my player and i've spent years getting the tags correct. I use my player as my primary music player. I listen in the car, and i stream over HTTP to winamp while at the office via the neet little web page i found here on the bbs. I would be pretty upset if i lost that data.

2. i've got a spare 120 gig firewire drive that i would like to use as the backup.

3. The util would need to do the following:

a FTP the files down from the player to a local folder by clicking the "backup" button. If this was the first time doing a backup, the util would FTP ALL the files from both drives down to a local folder. (might take a few days for large empegs)

b once files are downloaded, and i re-sync or re-backup, i would have the app only find the files that have been changed or added to the player since i last backed up. In other words, any files not on the local pc, transfer. That way, i wont have to re-FTP all the files which takes days.

c. once the files are sync'd, grab the ini file AND run a script to pull the play count and EQ settings from RAM to a file, THEN transfer that file. (i've done it before when upgrading my drive) There is an article about it in the FAQ section. (see this link: from the FAQ section )

I hope that would give me EVERYTHING that i would need to do a full restore WITH the play count, EQ settings, Wendy Filters, etc.


Can anyone add to this concept or help me start building it? I was thinking i could get a friend to help start building something in VP to do the FTP and file compares, but i'm not sure how i can run the RAM-to-file-then-pull-file script to backup the EQ and play counts. My programming expreience is only in ASP (little VP) and Windows only.

Basically, look at that FAQ link and i would want to automate that entire process, but only pull the CHANGED or ADDED files, so it wont take 3 days everytime i want to backup 100 gig.

Any ideas or help would be great.
_________________________
-CHiP

Top
#249611 - 16/02/2005 22:35 Re: empeg backup utility [Re: CHiP]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31600
Loc: Seattle, WA
Quote:
AND run a script to pull the play count and EQ settings from RAM to a file,

Minor semantic correction, you'd be grabbing those things from the scratch partition, not from RAM.


Things you need to think about when doing this:

1. Would need to work gracefully with either one disk drive or two.

2. Would need to work gracefully whether the player used a flat FID structure, or a FIDsifted structure.

3. Would also need a corresponding RESTORE utility. (A backup is no good if you can't restore it.)

4. If you could figure out how to trim it down to one-button operation, I think there's some kind of a bounty you'd need to arrange to collect from tanstaafl.

Truth is, these days, on Mk2 players with Hijack installed, it's theoretically possible (I think) to do all of this stuff with a cleverly written MS-DOS batch file. The only part that might not be do-able with the DOS FTP program is the "incremental" part of the backup you described. For that you need something like Rsync, and that requires a cygwin install on Windows PCs, if I recall correctly.
_________________________
Tony Fabris

Top
#249612 - 16/02/2005 22:43 Re: empeg backup utility [Re: tfabris]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
Quote:
you need something like Rsync


It would also require rsync to be installed on the player itself.
_________________________
Remind me to change my signature to something more interesting someday

Top
#249613 - 17/02/2005 02:58 Re: empeg backup utility [Re: andy]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Or just use ordinary mirrordir over FTP.

Cheers

Top
#249614 - 17/02/2005 03:01 Re: empeg backup utility [Re: CHiP]
tanstaafl.
carpal tunnel

Registered: 08/07/1999
Posts: 5549
Loc: Ajijic, Mexico
hope that would give me EVERYTHING that i would need to do a full restore WITH the play count, EQ settings, Wendy Filters, etc.


When you have it done, name your price!

tanstaafl.
_________________________
"There Ain't No Such Thing As A Free Lunch"

Top
#249615 - 17/02/2005 03:24 Re: empeg backup utility [Re: mlord]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31600
Loc: Seattle, WA
Quote:
Or just use ordinary mirrordir over FTP.

I didn't know that was an option! So FTPd in Hijack supports that?

Unfortunately that one doesn't seem to be an option in the MS-DOS version of FTP, so some third party solution would probably be needed for a Windows program, you couldn't just do it with a batch file.
_________________________
Tony Fabris

Top
#249616 - 17/02/2005 04:26 Re: empeg backup utility [Re: tfabris]
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
1. sure!
2. I think i have to learn what that is. i know what FID structure is, but I’m not sure what FIDsifted structure is.... ?
3. Restore... yes.. i figured i would be happy with just the backup. I can do the restore by hand no problem. I don’t want the restore to be something that I use all the time, only when i needed to test this thing out or actually restore from a failed system. If i can get it to work, I’m sure I can reverse the process for a restore. I wouldn't want to accidentally restore the wrong files.
4. huh? I must have missed a thread somewhere. 1 button is the idea, its what would make sense.

I would have to check into Rsync.

Top
#249617 - 17/02/2005 04:27 Re: empeg backup utility [Re: mlord]
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
i'll have to check into mirrordir.

Top
#249618 - 17/02/2005 04:30 Re: empeg backup utility [Re: tfabris]
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
I would use my developer's suite of tools for FTP. I have some FTP clients for VP apps that do a lot. If I’m going to do it, i want to stay away from a MS Dos batch file because that really doesn't do what i need. The tools i have for VB app programming is: IP*Works! V6 ActiveX Edition. Here are the Components that i can use with it.
_________________________
-CHiP

Top
#249619 - 17/02/2005 04:34 Re: empeg backup utility [Re: tfabris]
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
well, from what i can find about mirrordir, its Linux based.... I dont use any Linux machines. i'm 99% windows (1% empeg).

Top
#249620 - 17/02/2005 06:25 Re: empeg backup utility [Re: mlord]
andy
carpal tunnel

Registered: 10/06/1999
Posts: 5916
Loc: Wivenhoe, Essex, UK
Quote:
Or just use ordinary mirrordir over FTP.



Unless I am missing something about mirrordir it appears to be missing the one key feature that makes rsync so suited to this task.

With rsync running on both the PC and the player only changes to files are copied across the ethernet link between the two. Without rsync on both, using ftp, you will end up dragging every file across everytime you backup.

Perhaps mirrrordir does something similar, but I can't see how it can if it uses ftp to transfer the files. I'd check, but their sourceforge site is down.

I guess mirrordir could be checking the update time on the files, but is that reliable in our case ? Rsync uses hashes to see whether files have changed (though you can tell it to use file times instead).
_________________________
Remind me to change my signature to something more interesting someday

Top
#249621 - 17/02/2005 22:10 Re: empeg backup utility [Re: CHiP]
mcomb
pooh-bah

Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
Not to rain on your parade, but this sounds like a huge waste of time. Spend a half an hour getting rsync working which was made for this sort of task. If you can manage to write this program you describe setting up rsync should be trivial. Leave the rsync server running as a service on your PC and add a hijack menu_exec option to initiate the rsync from the empeg. Then you don't need to build a GUI and all you have to do is select rsync from the empeg and off you go.

-Mike
_________________________
EmpMenuX - ext3 filesystem - Empeg iTunes integration

Top
#249622 - 17/02/2005 22:22 Re: empeg backup utility [Re: mcomb]
tfabris
carpal tunnel

Registered: 20/12/1999
Posts: 31600
Loc: Seattle, WA
I don't think it's a waste of time... IF... it becomes a truly transparent one-button process that doesn't require you to jump through hoops to install it or get it working.

There are plenty of people who don't want to deal with getting rsync working or understanding command-line parameters. Those people would be very happy to see such a project work.
_________________________
Tony Fabris

Top
#249623 - 17/02/2005 23:16 Re: empeg backup utility [Re: tfabris]
mcomb
pooh-bah

Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
Quote:
I don't think it's a waste of time... IF... it becomes a truly transparent one-button process that doesn't require you to jump through hoops to install it or get it working.


OK, to throw in a slightly more useful two cents. AFAIK you won't get there using ftp as it doesn't have a foolproof way of recognizing changed files so your stuck pulling the full file over the wire every time. It also means you can't synchronize pieces of files (so if you change an ID3 tag you'll end up syncing the whole mp3). That means you are going to have to install something on the empeg to do incremental backups. This obviously is troublesome for a linux newbie or people wouldn't keep asking for a "one click" backup solution.

Anyway, given that ftp won't work you might as well use rsync or unison (similar to rsync, but does a 2 way sync) rather than reinventing the wheel. Neither of these will be trivial for a newbie to setup and both require cygwin on the windows side.

I still think he would be better off using an established tool and putting together a good write up that can be faq'd for the next person.

-Mike
_________________________
EmpMenuX - ext3 filesystem - Empeg iTunes integration

Top
#249624 - 18/02/2005 01:17 Re: empeg backup utility [Re: mcomb]
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
I don’t think this is a waste of time at all.... i think its worth at least a discussion...

ok, look.... if I can get rsync to work, which at this point i have never used, i might be ok, HOWEVER, I’m not sure how i would go about doing the dynamicbackup? My guess is a script of some sort. I don’t have much experience doing that, but i will give it a shot if i can start somewhere. You see, my thought is that if the dynamicbackup takes a minute to run, how does the script know to wait for it to be completed before running the next step of copying that dynamicbackup file over to the PC?

Remember, i want to do more then just ftp the FIDs, i want to back up the Play Count, Number of Plays and EQ settings found in the dynamicbackup (that data along with the tunes are more important to me then just backing up the tunes). So my solution would have to do the file transfer, then pull down the dynamic partition, and copy that file too. If this can only be done using a shell prompt, can i write a script file that i can just click on and run thoes commands at a shell prompt. (perhaps pulling open Term Serv to connect first). So I’d need to have the empeg connected via Serial AND Ethernet for all steps to complete.

I guess I’m just looking to get started but wanted to see other people's ideas and input first to see if there is an easier way, or to get other ideas going.
_________________________
-CHiP

Top
#249625 - 18/02/2005 09:47 Re: empeg backup utility [Re: CHiP]
peter
carpal tunnel

Registered: 13/07/2000
Posts: 4180
Loc: Cambridge, England
You can obtain the play counts and Wendy information from a stock empeg using just the normal emptool protocol. You can't, however, obtain the tuner presets, EQ settings, or bookmarks without installing third-party software on the player, such as Hijack.

And as you need Hijack's FTP server support in order to get the tuner and EQ settings, you might as well use it for everything, especially as clients are so easy to come by. You could, in theory, use the REST command to implement rsync over FTP, but in practice, backup isn't time-critical (leave it overnight) and so it's probably more effort than it's worth. If you don't mind a dedicated empeg backup, unconditionally retrieving the dynamic partition and then retrieving config.ini and all the fids if they've changed size or date, would be good enough and probably well under an afternoon's work in python. If you additionally want the PC-side version of the music to be usable and playable, you'd have to work a bit harder, and as the obvious representation for the playlists is a symlink forest, it's not clear whether it would make sense to do that to a Windows PC.

Peter

Top
#249626 - 18/02/2005 17:06 Re: empeg backup utility [Re: peter]
CHiP
enthusiast

Registered: 08/06/1999
Posts: 345
Loc: New Jersey, USA
I would not need to play the mp3s on the pc side. I have a dedicated drive just for the data. I would be 100% happy with a system that would (in as few clicks as possible) copy any changed FIDs, grab the ini, and grab the dynamic partition and simply store all that as files on the PC side. That's it. I just want something that is a recent back up so if the empeg HD dies, i can just put a fresh HD in, load the empeg software and then copy/restore the FIDs, ini file, and Dynamic Partition. I've done it before manually when i upgraded my HD and everything was restored just fine. I just want to make it easier so when i run this backup util, it will only grab what has changed since the last backup. The next best thing is to just copy everything by hand every so often when i feel like leaving the empeg for 3 days at home attached to my PC.

I'll check out the REST command. I have no experience in Python. Is there anyone out there willing to help build this util? i'm sure other empeg users would enjoy something like this.
_________________________
-CHiP

Top
#249627 - 18/02/2005 17:23 Re: empeg backup utility [Re: CHiP]
mlord
carpal tunnel

Registered: 29/08/2000
Posts: 14496
Loc: Canada
Don't worry about the FTP REST command -- it's actually not that useful for this particular application.

If you want to hack around installing/configuring it (and redoing again after any player s/w change), then rsync is the way to go for backing up anything in the filesystem.

But for me, it's just not worth the hassle. I'll just use mirrordir over FTP from a Linux box -- nothing special (other than Hijack) required on the player end. And also use FTP to grab the dynamic data partition as one largish file. That's it. Complete.

Cheers

Top
#249628 - 18/02/2005 19:26 Re: empeg backup utility [Re: CHiP]
mcomb
pooh-bah

Registered: 31/08/1999
Posts: 1649
Loc: San Carlos, CA
Quote:
HOWEVER, I’m not sure how i would go about doing the dynamicbackup?

The dynamic data is all stored in a partition on the empeg. The easiest way to back it up is to just grab a copy of the entire partition (it is small). You can do that with rsync or ftp by simply copying the appropriate /dev/hda entry. You don't need to run anything special first.

-Mike
_________________________
EmpMenuX - ext3 filesystem - Empeg iTunes integration

Top