It refused to half-upgrade

A piece of network equipment required a security update. The advisory indicated the current version was four releases behind. Active exploitation had been reported in the wild. The operator staged the update package and initiated the reboot.

The expectation was standard: cycle power, load the new image, return to service on the patched version.

It came back on the old one.

It had logged a single line explaining why. A second package - the driver for its own switching silicon - was still at the previous version, and it would not run a system whose two halves disagreed. So it had declined the upgrade entirely and booted the version it already trusted. Everything worked. Traffic forwarded. Management answered. It was simply still unpatched, and it had said so.

This refusal cost one wasted reboot and nothing else.

The value of the refusal

The valuable property was not that the device was clever. It was that the device had a condition it would not proceed without. It enforced that condition against an operator who was confident and wrong.

A half-applied upgrade would have looked like success. The management plane would have answered every probe. The forwarding plane would have been running an unsupported chip. Traffic would have dropped silently, the failure would have surfaced later as something else entirely, and the connection back to the upgrade would have been lost.

Systems that half-apply changes are how outages become mysteries.

If the device had accepted the mismatch, the operator would have spent hours tracing a routing loop or a dropped packet. They would have assumed the configuration was wrong. They would have assumed the network topology was the issue. The root cause would have remained hidden behind a functional management interface.

The refusal was a safety net. It caught the inconsistency before it mattered. It forced the operator to acknowledge the failure immediately. The log line was not an error message; it was a guardrail.

The condition of consistency

This behaviour illustrates a specific type of safety. It is not merely about detecting faults. It is about refusing to operate unless the entire system state is consistent. The operator uploaded one package. The device needed both. The device treated the missing driver as a hard stop.

This is distinct from a crash. A crash implies the system tried to run and failed. This was a pre-flight check that failed. The system knew it could not function safely. It chose inaction over incorrect action.

In complex environments, partial state is often worse than no state. A system that is partially updated presents a surface area that looks operational but behaves unpredictably. It answers queries while failing to process data. It creates a ghost in the machine.

The device protected the network from this ghost. It prioritised integrity over availability. In this context, availability meant nothing if the forwarding plane was unsupported. Better to be down and known than up and broken.

Generated work and atomicity

There is a parallel here to work that is generated rather than typed. The interesting question is not whether the thing can act. It is whether it has a condition it will refuse to proceed without.

When a model generates code, it often produces fragments. These fragments might compile individually. They might pass unit tests in isolation. But they may not fit together. A generated function might expect a data structure that was not generated, or it might rely on a library version that was not updated.

If the system allows these fragments to be merged without checking the whole set, it creates the same risk as the half-upgrade. The code looks complete. The build passes. The application runs. But it contains latent defects that surface under load or in specific edge cases.

We need interlocks for generated work. We need a mechanism that declines to apply a change unless its whole set applies. If a patch requires a database migration, the migration must succeed before the code is deployed. If a feature requires a configuration change, the configuration must be valid before the service starts.

Currently, we often rely on the operator to notice the mismatch. We rely on manual review to catch the inconsistency. This is slow and prone to error. It places the burden of safety on human attention.

What is still not solved

The refusal described above was the vendor's design. It was not something earned here. The device came with this safety net built in. The equivalent interlock for generated work largely does not exist yet.

It is easier to describe than to build. Defining the conditions for consistency across a complex system is difficult. The dependencies are often implicit. The versioning of internal components is not always tracked.

We can describe the property: a system should not proceed if its components are inconsistent. We can write tests that check for this. But we lack the runtime enforcement that the network device demonstrated. We lack the ability for the system to say "no" to itself.

This gap leaves us vulnerable to the half-upgrade scenario. We deploy changes that look complete but are fractured. We trust the build pipeline more than we trust the consistency of the result.

The lesson from the network equipment is clear. Refusal is a feature. A system that knows when to stop is more valuable than a system that always tries to finish. We must build that same discipline into our generated work. Until we do, we are accepting the risk of the ghost failure. We are accepting that the system might answer, but not work.

The device refused to half-upgrade. It was right to do so. We should aspire to the same behaviour.

← All writing