A while back I tweeted:

The Go developers keep breaking Go's self-tests when you have gccgo installed. Since they don't install it on almost any of their automated builders, I don't think they really care and so I'm done dealing with it myself. It's time to uninstall gcc-go.

I could file yet another Go bug report about this breakage due to gccgo that could as well have been filed by an automated machine, but I've lost all my interest in being a human machine for the Go developers. Building with gccgo present can and should be checked automatically.

(I don't actually use gccgo; I had it installed basically only out of curiousity.)

As a preface and disclaimer, I'm not picking on Go here in specific, because I think this is a general failure. It's just that I know the specific cases with Go and they make for a good illustration. The various incidents with gccgo are one of them, and then recently there was an issue when running tests in a terminal instead of in a situation where there was no terminal.

There's a saying in programming that if you don't test it, it doesn't work. Let me amend that saying; if you don't test it in your automated builders or CI, it doesn't work. This is true even if you have tests for it, and in fact especially if you have tests for it. If your CI is not running those tests, you are relying on the fallible humans making changes to remember to do testing in an environment where those tests will trigger.

(The problem with having tests for it that aren't being exercised by CI is that such tests are likely to give people a false sense of confidence. There are tests for it, the CI passed, and therefor the tests pass, right? Well, no, not if some tests are skipped by CI. And don't count on people to keep straight what is and isn't tested by CI.)

As these two examples from Go show, making sure that your CI triggers all tests is not necessarily trivial, and it's also not necessarily straightforward to guess whether or not CI will run a test. For that matter, I'm fairly sure it's not easy for your CI maintainers to keep up with what the CI environment needs in order to fully run tests, and sometimes doing this may require uncommon and peculiar things. It was perfectly reasonable to add tests to Go that ran to verify its behavior when it was talking to a terminal and it was perfectly reasonable for Go's automated builders and CI system to run builds not using a terminal (or a fake of it), but the combination created a situation where something obvious broke and wasn't detected by builders.