Ignoring the OS, USB really isn't all that great of a bus, and I doubt Linux can overcome protocol and design limitations. One issue I discovered is how bad it can be to mix a USB 1.1 and 2.0 device. Many sound devices are USB 1.1, so make sure to isolate them from any USB 2.0 devices on their own bus, otherwise there could very well be performance issues. This cropped up in my CarPC install, and on Windows at least, also exposed how much load device mixing like that put on the kernel drivers. I saw shoddy performance due to both bus load and CPU load.

If the system has firewire, you could put all the storage there, or find a firewire audio device, ensuring separation.