I know what you mean, I have this problem too. It never happened with v1.03, but beta11 has so much more advantages that I've learned to live with it.
Basically with b11 almost every time I skip a track I'd have to wait 2-3 secs before the tracks starts playing. This was not the case with 1.03 and I too blamed this on somesort of incorrect (or rather 'not optimal') hd caching technique.