A system that cannot tell you it is broken

Ask most teams how they would know a system had failed, and the answer describes something noisy. An alert fires. A page goes red. A number crosses a line. The whole apparatus is built around the assumption that failure announces itself.

The failures that do real damage are the quiet ones.

A feed stops arriving and the dashboard shows a flat line, which looks identical to a quiet week. A check passes because it was pointed at a path that does not exist, and a 404 is not an error to a script that only tested whether it got a response. A credential expires and the integration retries politely, every minute, for months. None of these ring a bell. Every one of them leaves you with a system that is confidently, continuously wrong.

Zero and nothing are not the same thing

This is the root of it. Almost every monitoring system in existence treats an absent value as a zero, and a zero as good news.

"No errors reported" and "no reports received" produce the same green tile. "Nothing suspicious found" and "the thing that looks for suspicious activity stopped running in March" produce the same clean report. The two states are indistinguishable on the surface and they could not be more different underneath, because one of them means the system is fine and the other means you have no idea whether it is.

A system worth trusting would refuse to collapse those two. Silence would be a finding in its own right - not an absence of findings, but a positive statement that something which should be speaking has stopped.

The tests that pass because they ask nothing

The same flaw wearing different clothes. A check written to confirm that a thing is unreachable will happily pass against a machine that was switched off, an address that was never allocated, or a path invented by accident. It reports success. It has tested nothing.

The tell is always the same: the check has no way to fail for the right reason. It cannot distinguish "the defence is working" from "there was nothing there to defend against". And so it passes forever, and it is trusted precisely because it has never gone red.

Any test asserting that something is blocked needs a companion asserting that something comparable gets through. Without the control, a green result is not evidence. It is just a green result.

What this would look like if it were taken seriously

A system built around this idea would look strange to anyone used to conventional monitoring, because it would spend most of its attention on things that are not happening.

It would know what it expects to hear from, and how often, and complain when a source goes quiet - even though nothing is technically wrong. It would treat "I have been running for six months and never once raised anything" as a claim requiring evidence rather than a badge of stability. It would alert on age rather than on count, because a growing queue is throughput and an old item is neglect, and only one of those needs a human.

Most of all, it would be able to answer the question that matters and is almost never asked: what would I currently be unable to detect?

That question is uncomfortable, which is why it goes unasked. The honest answer usually includes several things nobody realised had stopped.

The cost of getting it wrong is measured in time

A loud failure costs you an outage. Everyone knows immediately, everyone responds, and the damage is bounded by how fast the problem gets fixed.

A quiet failure costs you the entire period during which you believed you were covered and were not. There is no incident, no response, and no bound. The clock is running the whole time and nothing is counting it.

Which is why the design goal is not "raise the alarm faster". It is something more awkward and much more useful: build systems that cannot fail silently, then assume they did anyway, and go looking.

← All writing