Add to that the fact that, if you move it away from the holes, less air actually gets blown out of the casing. When I moved it away from the holes, I couldn't feel any air coming out of them (fan was oriented to move air up and out of the case).

Yes, well, there is a very good reason for that. If the fan casing isn't hard up against the lid then you have to seal the gap up with something don't you. Otherwise you just end up stirring up the air inside the case, especially once the gap is big enough that the resistance to flow through the gap is much lower than the resistance to flow through the holes, which is all a bit pointless really isn't it.