That's excellent food for thought. That scenario is indeed possible. This is cutting to the heart of my question, this is exactly the kind of scenario I'm talking about and which really needs a clear method for addressing it.

What really should happen is that there should be an option in Nunit test runners to flag multiple repeated "inconclusives" which occur across multiple test runs. Or some other way to flag it perhaps with a different kind of assert.

There's already something built into Team City which allows you to set a failure threshold. For instance, you could say it's OK for up to 3 integration tests to fail before it starts putting up the red flag. One of our test suites is configured that way, but then there was the time that a dev checked in a regression which caused exactly 3 of those tests to fail...

I'm looking for industry standard best practices for this exact sort of thing. Clearly we're not the only company who runs into issues like this. I'm having trouble finding specific examples of how you handle these situations with common tools.