I'm not getting as many good hits as I'd like from Google on this topic. I've got a few, but there's not a lot of detail in the areas I'm looking for.

We keep our unit tests and our integration tests very separate. The unit tests are truly atomic, and if someone breaks a unit test, everything stops until it's fixed. That's as it should be.

Our automated integration tests are a bit more wibbly wobbly (as they should be, I suppose). Some of our integration tests depend on third party resources which might have temporary down time, for example. Sometimes, our integration tests fail for what seems to be no good reason. It's not a huge problem, but we're looking at taking some time to focus on improving the reliability of our automated integration tests. I have a list of things that I think are good practices, but they're mainly from my own mind rather than from any sort of industry-standard Best Practices guide. I have yet to find a Best Practices guide in a google search which gets into the level of detail that I'm getting into. I'm wondering if anyone has seen any really good guides for this on the internet that I'm missing?

Here's an example of my level of detail, and I'm not seeing anything with this kind of level of detail in any of the Best Practices lists that are coming up in my google searches:
"If test is clearly failing because third-party resource is temporarily down, which is obvious in the failure output: Suggested plan of action: Test author should add a pre-check for the resource in the test, and Assert.Inconclusive if the resource is down (and ONLY if the resource is down)."

Track the wibbly-wobbliness of the integration tests over time. You'll find a bunch that are consistently inconsistent. Those tests need more love than the others.

Those tests probably have race conditions in them. Fix the race conditions.

When fixing race conditions, prefer waiting for an event over simply sleeping. In the worst case, the sleep won't fix anything (because build agents are always slower than you think), and in the best case it's slowing down an already slow test suite.

If you can't wait for an event, prefer polling for status (with short intervals) over simply sleeping.

Consider adding event / status polling endpoints to your product. If you can't make them secure, make sure they are turned off in production. If you can make them secure, document them and tell your users about them. They'll find them useful, too.

For implementing event endpoints, if you're using HTTP, remember that server-sent-events are a thing. There are client implementations in all popular languages. You don't need to use websockets.

Those are fantastic pieces of advice. I agree with every single one. For the most part, we're doing all of them to some degree or another, but there is always room for improvement.

What if the resource is down permanently, but only from the view of the tests? Maybe someone broke the firewall rules on the build agent, or maybe your test API account expired?

You'll end up never running those tests, and you'll ignore the inconclusive results, and now you're running code in production that's never getting tested.

Maybe invent another assertion that fails on Mondays, but is inconclusive the rest of the week...? I can't come up with a good name, though.

That's excellent food for thought. That scenario is indeed possible. This is cutting to the heart of my question, this is exactly the kind of scenario I'm talking about and which really needs a clear method for addressing it.

What really should happen is that there should be an option in Nunit test runners to flag multiple repeated "inconclusives" which occur across multiple test runs. Or some other way to flag it perhaps with a different kind of assert.

There's already something built into Team City which allows you to set a failure threshold. For instance, you could say it's OK for up to 3 integration tests to fail before it starts putting up the red flag. One of our test suites is configured that way, but then there was the time that a dev checked in a regression which caused exactly 3 of those tests to fail...

I'm looking for industry standard best practices for this exact sort of thing. Clearly we're not the only company who runs into issues like this. I'm having trouble finding specific examples of how you handle these situations with common tools.

Nice article on this topic:

https://samsaffron.com/archive/2019/05/15/tests-that-sometimes-fail