Hello All,
I was wondering if any of you have or know of a program that can equalize the volume of all your mp3 songs? I have just started to record MP3's for my Empeg, once I get it. As it stands, I have 1300 songs so far, all recorded on the same pc but the volumes of older cds vs: newer cds and those vs: homemade cds from private recordings, varies. I know this is how the individual cds are, but I was hoping for some sort of mp3 volume adjusting software that will batch process all you mp3's to a relative volume level. Any ideas?
I have heard of programs that will batch transform your cds from a high kps to a lower one, but it does not affect the volume.Thanks...
Eryk

This was discussed recently on the "Wish List" thread.

Equalizing the volume of your songs is a process called "Normalization". This can only be done at the beginning of the process, when the file is still in .WAV format. This option is available in most ripping software so that it happens automatically for you (provided you enable the option).

Because of they way MP3s are encoded, the only way to change the volume of an already-compressed song would be a lossy process: De-compress the MP3 to a .WAV, normalize the .WAV, then re-encode the .WAV a second time. If you re-compress something that's already been compressed once, you might induce audible artifacts. So it's not ideal. In fact, I don't know of any program that does this for you.

Usually, the only way to solve the problem properly is to re-do your collection from the original CDs, while enabling the normalization option in your ripping software. I'm in the same boat as you are: I have a lot of albums that I encoded without normalization, before I knew it was important. I'm not prepared to re-rip everything now.

My suggested solution (on the wish list board here) is to have the MP3 player software auto-adjust the volume of the next song before it begins playing it. It could do this by peeking ahead at the next song and calculating what relative volume it should be played at. This would be a very serious undertaking and it's not trivial to implement at all. The Empeg folks are pretty busy implementing more important features and fixing bugs, so it's highly unlikely we'll see this any time soon (if at all).

There is a new tag specification for MP3s (called ID3v2) that allows a "relative volume" field to be added to each file. At the moment, the Empeg can't read these kind of tags, so that's not available to us Empeg users, either.

One thing that might help is this: Very soon, the Empeg will have the ability to link a specific song to a specific EQ preset. This is on their "to do" list. You could have some EQ presets that are louder than others (by boosting more frequencies than you cut). For me, this would work perfectly, because my older albums also could use some extra EQ trimming to sound good compared to new albums. These are the same songs that need to be normalized anyway.

Ahhh..ok. Thanks. I found the normalization check box and, as you had said, it was not on. I turned it on and will have to , some day, start the process of re-recording my MP3's. At least all the new ones I record will be set.

Thanks for this info.

Technically, you can in fact change the normalisation while it's in .MP3, but that requires decoding it, scaling all the coefficients, then recoding. This doesn't involve conversion to .WAV at any point, so it's lossless. And no, I have never seen anything that does this :)

I wonder if patents allow for such a program, even. Anyone a qualified patent lawyer? :)

- John.

(The above may not represent the views of empeg :)

If this is true, I would like to enter a detailed discussion of this topic with someone knowledgeable about it. I might be interested in writing a program that does this.

I was under the impression that the MP3 frames were encoded in such a way that prevented this kind of manipulation.

What can you tell me about this? Can it be done simply by locating the proper bytes and altering them? Or would a lot of math be involved? What do you know?

> What can you tell me about this? Can it be done simply by locating the proper bytes and altering them? Or would a lot of math be involved? What do you know?

There'd be quite a lot of work involved here, and a bit of math(s) too. First, you need to know the mp3 format. You then decompress (huffman coded I think) the DCT terms for each frame to get the frequency components. Multiply all the components by a scale factor (must be the same for every frame or you'll get glitches between them). Then compress the terms again with huffman and write out an mp3 stream.

This sounds complex (it is) but it'll take far less time than decompress/recompress via PCM data - definitely something that can be done real time. Not a simple job though.

- John.

(The above may not represent the views of empeg :)

But isn't there more to it than just multiplying by a scale factor? Doesn't changing the volume of the frame data change the characteristics of the compression algorithm?

I know the file format, but I don't understand the arcane details of the compression/decompression algorithms. Once I'm past the frame header, I'm lost. That's why I'm asking.

The "compression algorithm" at the bit level in MPEG audio layer III is really just the Huffman coding and the use of a bit reservoir. Any decoding and recoding here possibly runs the risk of changing the length of the main_data for the frame, and this could cause problems if the bit reservoir is run short.

What might be interesting to try instead is to manipulate the global_gain value for each of the two granules in each frame. Since this is a constant-width field in the frame's side information, modifying it would be trivial. However, you would still need to decode the entire stream to determine an appropriate scaling factor. Also, I have no idea whether the effects of changing this value are really acceptable in practice -- it's only an 8-bit value, and it's not clear to me how much the other coded information is predicated on it.

Another option it seems to me is rather than modify the audio stream itself, modify the decoder to perform the scaling in real time while the audio is playing. This assumes the player can know in advance by what factor to scale, but people have talked about some ways to accomplish this.

This does sound like an interesting project for someone to pursue. As far as patents are concerned, to my knowledge most of them concern the psychoacoustic model and the encoding process; what I think is being proposed here is just some bit twiddling on the resultant bitstream.

-v

...for the sake of the empeg, wouldn't make more since to simply do it on the player side? Maybe even as a plugin that pre-analyzes the next song, and lowers or raises the volume accordingly...

This would mean that those of us with 100 cd's on our setups don't have to spend time looking for our originals...

...proud to have one of the first Mark I units

Unfortunately modifying the global_gain alone as I suggested doesn't seem viable; some tests I did suggest it will result in utterly unrecognizable audio.

Any in-place modification of the MP3 stream would probably need to recode the entire frame (without overrunning the bit reservoir.)

-v

What might be interesting to try instead is to manipulate the global_gain value for each of the two granules in each frame. Since this is a constant-width field in the frame's side information, modifying it would be trivial.

Now we're talking!

You seem to really know your stuff. Can you help give me some specifications as to how I might be able to locate these bytes in a standard 128kbps 44khz MP3 file? Are they part of the frame header? I've already got some code that dissects the frame header a little bit and performs some simple operations based on it. I'd love to experiment with it and see what I can come up with.


However, you would still need to decode the entire stream to determine an appropriate scaling factor.

Yes, of course. If you wanted to go to that kind of trouble. Although a simpler program could just allow you to manually enter the scaling factor, then preview your changes by ear.

If it worked at all, anyway...


Another option it seems to me is rather than modify the audio stream itself, modify the decoder to perform the scaling in real time while the audio is playing. This assumes the player can know in advance by what factor to scale, but people have talked about some ways to accomplish this.

Exactly. As I've said before, a playback-only normalizer sounds technically feasible (but not easy).

Oh, nevermind, then. :-)

Damn, had me excited for a minute there.

...for the sake of the empeg, wouldn't make more since to simply do it on the player side? Maybe even as a plugin that pre-analyzes the next song, and lowers or raises the volume accordingly...

Yeah, it's in the "Wish List" thread. I put it there. But like I said, it's not trivial to implement something like that. It would take a lot of time for a programmer to implement it properly. The Empeg programmers have a lot on their plate as it is right now.

Even if it could be implemented, It would require a significant amount of CPU time, and it would also double the amount of disk access the player performs.

Wait a minute... would it?

If the player is already precaching the next song anyway (I know it does), then maybe it wouldn't increase disk usage. Well, at least for short songs. For symphonies and one-track rock operas, it would require quite a bit of extra disk access. But for pop songs, you could probably decode and analyze most (all?) of the next track without hitting the disk again. Hmm...

I would thing that it could be made as a program that run ON THE EMPEG (so that you don't have to actually re-upload your songs) that maybe adds a comment in every mp3 file on the empeg w/ the normalization value.. The player software could then just read this comment to get the right value... and future songs normalization values could be added using emplode. That way you could just run the program for an afternoon or so, as opposed to having to reupload every song over again.

...proud to have one of the first Mark I units

In case you're still interested in playing around with it (I may have goofed in my haste to test), I'm attaching a code excerpt which decodes the layer III side information from the bitstream, immediately following the header (and CRC, if there is one.) This isn't the entire frame, as the main_data portion can be located before the header and continues after the side information. The main_data_begin is a negative-offset pointer from the beginning of the frame to the beginning of the main_data. Code to decode the main_data (which includes the Huffman codes) is not included.

The bit_read() routine is supposed to read the number of specified bits from the stream and return their value. The code should be enough to show you where the global_gain field is, among others. To understand all the fields though you really need to have a copy of the MPEG audio spec.

I don't know why the BBS isn't registering my attachment, so get the code from here.

Cheers,
-v

Thanks! I've downloaded the code and I'll play with it when I get a chance.

Interesting idea.

What you're saying is... write a Linux program to run on the Empeg... It runs through your Empeg's files, analyzes their volume levels, and adds the appropriate relative-volume command to the music database for each song.

Simple. Elegant. Relatively easy to implement (when compared to a dynamic system). Doesn't affect caching or CPU cycles during playback. The resulting data could simply appear as another field on the song's property sheet in Emplode, allowing you to manually tweak the numbers if you choose.

It could even happen automatically at upload time, as part of the synchronization process. After the rest of the synch is complete, any songs on the Empeg's hard drive that don't yet have the field tagged would get scanned. The first time a user runs this new version of the software, it would run on all their songs, and it would take a long time, but subsequent runs would only analyze new uploads, so it would be quick.

Even then, you could optimize the process by only analyzing some of the frames. Say, every other frame or every third frame. I'd bet you'd get the same results. A little experimentation could give you the optimal balance between how accurate the analysis is compared to how much time you spend decoding frames.

The initial version could simply run the analysis as a batch job after the songs have been uploaded. Future versions could speed up the process by performing the analysis as the song data is being uploaded, resulting in practically no extra synch time wasted at all.

Of course, it would have to be a selectable option in Emplode-- Users who have already pre-normalized their songs should be able to turn off the option.

But most importantly, this is an elegant solution because it would be transparent to the end-user. They wouldn't need to worry about anything more than whether or not to select this menu option. Just select the menu option, and the Empeg magically normalizes all the songs on the next synch.

I like it. I really like it.

"Mac", are you reading this? What do you think?

exactly... Songs that haven't already been normalized would have a value equivilant to 0, and songs that have been normalized would get bumped up or down depending... And the best part is, the original file would be left alone, and the user could change the setting for that perticular song (through emplode?) if he or she chooses...

...proud to have one of the first Mark I units

Well, the actual coding would require that the "not yet normalized" flag be something other than a 0 in the field, since it's possible that a given song might not need any change in volume. I don't know how the values are stored in the music database, so I don't know how it would be flagged. But yeah, that would be the basic idea.

I'd really love to hear what Mike Crowe ("Mac" here on the BBS) has to say about this.

Oh, and sorry for carrying on such a technical discussion on the "General" message board. :-)

I'd really love to hear what Mike Crowe ("Mac" here on the BBS) has to say about this.

*Looks around*
Who? me?

Oh, ummmm.

After minimal thought it does look feasible to normalise songs during download or to add a normalise option to emplode which works rather like the "play on empeg" toolbar button.

An alternative (suggested by someone else some time ago) is to analyse the song as it is played the first time through and then store a normalisation value ready for next time.

I think that automatically normalising the next song before it is played is rather too complex. Dealing with track skipping and the fact that the whole of the next track won't actually be cached would require the disk to be spun up much more than is desirable.

--
Mike Crowe
I may not be speaking on behalf of empeg above :-)

An alternative (suggested by someone else some time ago) is to analyse the song as it is played the first time through and then store a normalisation value ready for next time.

I was going to recommend the same thing. Not only for normalisation, but also for the "Time Remaining" counter. The play time could just be stored in another field of the database.
I wouldn't mind having to play the song first before I have this info the first time.


_~= Dearing =~_
"WAY too happy about having #99."

It would certainly be nice to store information calculated
during the playing of a song for further reference.

I can think of at least four other other applications of this
that would be nice (if the player had hooks so we could add
modules in).

You could even have "persistent" volume adjustments made by the user during
the song, so that next time it is played, the volume adjustments are
automatically made at the same times.

However, is it feasible? I thought all partitions were
mounted read only.

How difficult would this sort of thing be?

Also, I seem to recall seeing somewhere that we would be able to
retrieve statistics from the player as to how many times a particular
track had been skipped, and that sort of thing. Is that implemented
yet?

Richard

Who? me?

Yes, you.

After minimal thought it does look feasible to normalise songs during download or to add a normalise option to emplode which works rather like the "play on empeg" toolbar button.

Cool! Sounds great! So when can we see it implemented? Day after tomorrow?

Just kidding. I don't know about everyone else, but there's other stuff I'd rather see sooner, such as linking different EQ presets to specific songs, and having the EQ's automatically switch personalities depending on whether it's plugged in at home or in the car. I'm sure you've got your own lists of priorities anyway...

I am a little bit confused about this "Normalization" business. What exactly is normalization supposed to accomplish? It sounds like you are attempting to have all the music play at the same volume, but this would not be right. I don't expect a Haydn string quartet to make as much noise in my car as, say, Metallica. Some music is supposed to sound louder than other music.

Would you please expand on this normalization concept a bit for me?

Thanks...

tanstaafl.



"There Ain't No Such Thing As A Free Lunch"

I don't expect a Haydn string quartet to make as much noise in my car as, say, Metallica. Some music is supposed to sound louder than other music. Would you please expand on this normalization concept a bit for me?

You're right, some music is supposed to sound louder than other music.

However, in the case of CDs, songs that are supposed to sound at the same volume might not be that way, depending on how the CD was mastered.

For instance, old CDs that were made in the heyday of vinyl don't push the dynamic range of the format very much. They were mastered for the dynamic range of LPs, and they don't get as loud as the CD will go. But new CDs do, because they've been mastered from the ground up with the CD format in mind. Compare any CD from the 80's to any CD created within the last few years. All the recent CDs are much louder, by more than just a few decibels.

This even applies when you're comparing the exact same song. For instance, you might have an old CD in your collection that's got the exact same song as a new remastered version of the album. The old CD will be MUCH quieter. Often, the songs get normalized when they create an artist's "greatest hits" album so that the songs fit together as a collection better.

Normalization isn't an issue for regular CD players because you'll tend to listen to one album at a time, and the whole album will be at its proper relative levels. But when you have something like the Empeg, you could be listening to a fifteen-year old track right before a recent track. What happens is that you turn up the volume for the old song, then when the new song starts, it's THIS F***ING LOUD and you have a heart attack scrambling for the volume control and hope that you didn't just blow your 200-dollar speakers.

Now, obviously this applies to pop music more than classical music. However, the same issues exist: older albums won't ever hit the peak levels on the CD, while new ones will. Remember that normalization doesn't just artificially make a song louder... the idea is that you measure the peaks in song, then you normalize the volume so that one song's peaks are pretty close to the same as the last one's. Even quiet songs can have loud peaks.

Radio stations have known this for years, and they use compressor/limiters to normalize their music. They also compress the audio to fit the dynamic range of FM radio, and to make the songs blend better with the commercials.

The Empeg, by its very nature, clearly shows how much difference there is in the mastering techniques from album to album. The equalization, compression, and overall volume of a CD will vary widely from album to album. That's why we need features like per-song EQ and normalization.

It sounds like you are attempting to have all the music play at the same volume, but this would not be right. I don't expect a Haydn string quartet to make as much noise in my car as, say, Metallica. Some music is supposed to sound louder than other music.

You are right. That's why I like the idea of having some sort of "global gain" attribute stored with each song in the empeg's database, with no changes to the actual music data. There could be a function in Emplode to analyze the average volume of a song and then make a proposal for that attribute, but the user would be able to manually change it, or not have it set at all.

In my eyes this combines the advantages of many of the proposals here: rather easy to implement on the empeg side, no distortion due to re-evaluating the music data, total user control over the result.
As an extra bonus, this attribute could be taken from the corresponding ID3v2 Tag value if it exists...

Regards
Daniel

There could be a function in Emplode to analyze the average volume of a song and then make a proposal for that attribute, but the user would be able to manually change it, or not have it set at all.

Yup, that would be the perfect solution, for all the reasons you mentioned.

One important issue with it, though... (Oh, Mike, are you still reading this thread?)

You suggest that Emplode analyze the song. I disagree. I believe the analysis should run in Linux on the Empeg itself (and be triggered by Emplode during the synch process). Reasons:

1) There's no MP3-decoding code in Emplode (as far as I know), they'd have to write a decoder into Emplode before it could do it. On the Empeg itself, the decoder is already there.

2) When you upgrade to the software version that contains this feature, it would be able to scan all the songs that you've already uploaded to the Empeg. You wouldn't have to re-scan the files as they exist on your computer. For some people, the MP3s in their emplode might not have a 1:1 correspondence with the ones stored on their computer. I know that I personally have altered the directory structure on my computer since uploading the bulk of my songs to the Empeg.

3) The normalization values could get written directly to the Empeg's music database immediately as the scan was progressing. More efficient than uploading these fields after-the-fact.

4) If it was done as part of (or at the end of) the synchronization process, then it becomes totally transparent to the end-user.

Mike? Comments?

I take this back.

It turns out the Layer III global_gain field can be modified after all, with the net effect of increasing or decreasing the overall gain of the output just as you might expect. To be consistently uniform, it needs to be offset by the same amount in every channel in every granule in every frame (two channels per granule in stereo streams, and two granules per frame.) The code I posted earlier should provide enough information to locate the field(s) within a frame.

Sorry I goofed when I tested this before. I have no idea what I was doing.

I might even have enough code to compute an appropriate offset value in order to normalize the output... if anyone still has any interest in this, let me know.

-v

if anyone still has any interest in this, let me know
Verement, you've just made the understatement of the year! (so far)

Remember that this thread started out with a post from calseeor that said(quote):
As it stands, I have 1300 songs so far, all recorded on the same pc but the volumes of older cds vs: newer cds and those vs: homemade cds from private recordings, varies.
( . . ) I was hoping for some sort of mp3 volume adjusting software that will batch process all you mp3's to a relative volume level.
(unquote)

I bet we all have a dozens of playlists that we'd like to have normalised among the hundreds that are recorded right.!
Yes Yes Yes, we're interested. Especially if you can make it to run in the empeg box.

Sorry I goofed when I tested this before. I have no idea what I was doing.
We'll forgive, once you deliver

Henno
# 00120

I haven't had a chance to look at the code you posted before. I downloaded it to my hard disk, and it's on my "to do" list...

Of course we'd be interested in it. And I'll tell you what: If you could come up with a simple Windows front-end to select (or group select) MP3 file(s) and batch-process their gain fields, you could shareware-sell that puppy over the internet. There would be a big demand for such a thing.

One interesting issue:

I've been doing a little experimenting, and I've discovered that peak-detection normalization is not going to work "automatically", since many of the albums that I consider "quiet" are actually already peak-normalized. It's just that newer albums are compressed more, with the quiet bits being closer to the peaks. If you increase the global gain on a track that is already normalized, you will clip the peaks just a little bit. For the albums that I want to increase, though, I don't think the effect would be noticeable. If you could come up with a simple test program, I have a few tracks in mind that I could try it on (ones that sound VERY quiet overall except for a couple of very large peaks) and see if it's a viable to do that.

The other option is to leave the old albums unchanged, and only reduce the global gain on the newer albums. That would have the same net effect, but without the peak clipping.



Tony Fabris
Empeg #144