Industrial reliability: change the denominator

15 Jul 2026

I was building the operator interface for a force-controlled industrial robot — an arm that presses a powered tool against a workpiece with a programmed force and moves it along a path. My client asked me the best question anyone has asked me in years: "The problems that hit us on the floor will be the ones we didn't think of — that's what 'didn't think of' means. But every reliability estimate uses 'the things we thought of' as its denominator. So how do you ever guarantee industrial reliability?"

Why "review harder" never converges

Review finds the bugs you can imagine. The bugs that hurt you are, by definition, the ones you couldn't. So you review again — the second pass finds a few more imaginable ones, the third fewer. The sequence doesn't converge to zero; it converges to "everything I was capable of imagining," which is not the same thing at all. On that project I'd already run five rounds of review, my own plus independent cold-eyes passes, and every round still found real defects. That's not a sign the code was unusually bad — it's a sign that "review harder" is a treadmill. If round five still finds bugs, round six will too, and you can never stand in front of a moving machine and say "there are no more."

Counting bugs is a losing game: the denominator is infinite and unknowable. The way out isn't to count better — it's to change what you count.

Change the denominator

The whole idea in one sentence: stop trying to guarantee "my software has no bugs"; guarantee instead that "even if my software is completely, arbitrarily wrong, the consequences are bounded." The first is impossible. The second is achievable — because it swaps an infinite denominator for a finite one. You cannot enumerate your bugs, but you can enumerate the kinds of harm a bug is able to cause. For a machine like this, the entire space of bad outcomes fits in five buckets:

• A person is hurt. · The machine or the workpiece is damaged. · Saved data is lost. · The system halts or the screen freezes. · The job comes out wrong.

It doesn't matter how many bugs you have or how exotic — the damage they do must land in one of those five. There is no sixth kind of bad day. So the question transforms from "how many bugs did I think of?" (denominator ∞, never done) to "for each of five consequence classes, do I have a backstop that does not depend on my own code being correct?" (denominator 5, finite, checkable, done). That last clause is load-bearing: a backstop written in the same codebase as the bug is the same author grading their own exam. The guarantee has to bottom out in a layer the buggy software cannot author or weaken.

The finding that proved it to me

In a later phase, the interface would send the operator's chosen parameters — force, speed, path counts — down to the robot's motion controller. The transport for that was vendor-supplied boilerplate that shipped with the platform template. It had sat in the tree, unused, since day one. Five rounds of review glided right over it — correctly, because it wasn't executing yet; line by line, nothing was "wrong." Then I stopped reviewing and asked the consequence question instead: assume this transport is compromised — what's the worst it can do, and what stops it?

The boilerplate built the message to the controller by pasting the parameters, as text, into the middle of a program the controller would then execute — trusting a standard serializer to keep them contained. But that serializer escapes some quote characters and not others. Which means a parameter value containing the right character could close the quoted string early and turn the rest of itself into live instructions for the robot. Now follow the path: one of those parameters was a name — a label the operator types, or worse, a label riding in on a backup file imported from a USB stick. An untrusted file on a removable stick → arbitrary motion on an industrial arm with a powered tool spinning on the end of it.

That defect was invisible to five rounds of line-by-line review — not because the reviewers were careless, but because by the "is this line correct?" test the line was correct, and it wasn't even running. It became visible instantly the moment I switched from "find the bug" to "assume it's compromised; where's the backstop?" There was none. The only thing between an untrusted file and a moving machine was my own code being perfect — the one thing I'd just spent five rounds proving I could not promise.

What the backstops actually look like

• Re-check the limits on the far side, in a layer you don't author. My interface clamped force and speed to safe limits — but that clamp lives in the same software that might send the bad value. So the controller clamps them again, independently, in its own program that I don't write. Now the interface can send 999 and the machine still uses 40. The far-side clamp is the authority; the near-side one is a courtesy. This one line is the difference between "safe if my code is right" and "safe."
• A physical envelope the software cannot reach past. Most industrial controllers can enforce a geometric box, configured once behind a safety password, that the motion system itself refuses to leave. Set it tightly around the work cell, and it no longer matters what coordinate my software computes — correctly or insanely — because the safety layer, which my application cannot author, weaken, or even see, stops the arm at the wall. It's the only real backstop for a bad-coordinate bug, where clamping is meaningless because a position isn't a number you can min/max.
• Staged unlock — turn "didn't think of it" into "saw it happen, safely." You don't go from "software controls the machine" to "full force on a real part" in one step. You pass gates, each bounding the consequences of everything you haven't discovered yet: Dry run — the machine doesn't move; the software runs for real and logs every command it would have issued (a wrong parameter costs a line in a log). Air cut — real motion, tool off, path lifted above the work; a wrong path draws a shape in the air. Scrap material — real force on a piece you'd happily throw away. Then production. The bug you didn't think of is still in there at every stage — that's the point, you never removed it — but reality reveals it at the stage where its consequence is bounded.

That's what industrial reliability actually is. Not "the machine won't have problems." It's: when the machine has a problem — and it will — the problem surfaces in the air, above the work, with the tool switched off, and nobody gets hurt.

The part nobody says out loud, and should

Here's the sentence I think every honest vendor of anything that moves should be willing to say: we are not building toward "no bugs." We know there will be bugs — including ones we'll never find. We're building so that when a bug fires, it's caught, or bounded, or physically impossible to turn into harm. That sounds like an admission of weakness; it's the opposite. The person who promises bug-free software is naive or selling. The person who shows you the gates, the two independent clamps, and the box the arm can't leave has thought about the day it goes wrong — the only day that matters. "Bug-free" is a marketing word; "consequence-bounded" is a design.

Written from a real force-controlled robotics integration; all client, vendor, and product specifics removed. The method is general — it applies to anything where a software failure has physical or irreversible consequences, from motion control to payments to deploys.

Where this comes from

None of this is my invention — it's the established discipline of safety engineering, put in plain words:

• "Testing shows the presence, not the absence, of bugs." — Edsger Dijkstra, 1969. The root of "you can't test for what you didn't foresee."
• Nancy Leveson, Engineering a Safer World (MIT Press) — why safety is not the same as reliability, and why you constrain the system rather than try to perfect every part.
• IEC 61511 — Layers of Protection Analysis: the industrial standard behind independent backstops (a safety layer must keep working even if the control system fails).
• The "Swiss cheese" model (James Reason) — why several independent layers catch what any one of them misses.

← All notes