Industrial reliability: change the denominator

Why “review harder” never converges

Review finds the bugs you can imagine. The bugs that hurt you are, by definition, the ones you couldn’t imagine. So you review again. The second pass finds a few more, the third fewer. The count doesn’t fall to zero; it approaches “everything I was able to imagine”, which is a different thing. On that project I had already run five rounds of review, my own plus independent fresh-eyes passes, and every round still found real defects. That doesn’t mean the code was unusually bad. It means “review harder” never finishes. If round five still finds bugs, round six will too, and you can never stand in front of a moving machine and say “that’s all of them.”

Counting bugs can’t be won, because the denominator is infinite and unknown. The way out is to change what you count.

Change the denominator

The idea in one sentence: stop trying to guarantee “my software has no bugs” and guarantee instead that “even if my software is completely wrong, the consequences are bounded.” The first is impossible. The second can be done, because it replaces an infinite denominator with a finite one. You can’t list your bugs, but you can list the kinds of harm a bug could cause. For a machine like this one, every bad outcome falls into one of five groups:

• A person is hurt. · The machine or the workpiece is damaged. · Saved data is lost. · The system halts or the screen freezes. · The job comes out wrong.

However many bugs there are, and however strange, the damage must fall into one of those five. There’s no sixth kind of bad day. So the question changes from “how many bugs did I think of?” (denominator ∞, never finished) to “for each of the five consequence classes, is there a backstop that doesn’t depend on my code being correct?” (denominator 5, finite, checkable, finishable). That last condition matters most. A backstop written in the same codebase as the bug is the author grading their own exam. The guarantee has to rest on a layer the buggy software cannot write or weaken.

What a consequence-first review turned up

Later in the project, the interface had to send the operator’s settings down to the robot’s motion controller. I had already reviewed my own code line by line five times. This time I asked the consequence question about everything between the operator and the motion, including code I hadn’t written, such as the integration code supplied with the vendor’s platform: assume this path is hostile. What’s the worst it can do, and what stops it?

It turned up serious security problems, and they weren’t unique to this project: the same class of flaw shows up in integration code from many vendors. In short, input from outside the system could end up controlling far more of the machine than it ever should. I’m not going to say where or how, because code like this ships in other people’s machines and the details would help someone attack them.

Five rounds of line-by-line review had missed it, and not through carelessness. Line by line, the code was correct, and part of it wasn’t even running yet. It became obvious the moment I stopped asking “is this line right?” and asked “if this is hostile, where’s the backstop?” There wasn’t one. The only thing between untrusted input and a moving machine was my own code being perfect, which I had just spent five rounds showing I couldn’t promise.

What the backstops look like

• Close off the whole class, not just the instance. Patching the one spot would only fix that spot. The durable fix is structural, and it’s what I now apply to every path that reaches a machine. Values from outside the system travel only as data, never as part of anything the controller executes. Every value is checked against an allow-list and hard limits where it enters, and checked again by the controller before anything moves. Anything imported from outside is untrusted until it has been fully validated. When input can only ever be data, and is checked at every boundary it crosses, this kind of attack has nothing to work with, wherever it comes from.
• Re-check limits on the far side, in a layer you don’t write. My interface clamped force and speed to safe limits, but that clamp lives in the same software that might send a bad value. So the controller clamps them again, independently, in its own program that I don’t write. The interface can send 999 and the machine still uses 40. The controller’s clamp is the one that counts; the interface’s clamp is a convenience. That one line is the difference between “safe if my code is right” and “safe”.
• A physical envelope the software can’t reach past. Most industrial controllers can enforce a geometric box, set once behind a safety password, that the motion system itself refuses to leave. Set it tightly around the work cell, and it no longer matters what coordinates my software computes, right or badly wrong, because the safety layer stops the arm at the boundary. My application can’t write, weaken or even see that layer. For bugs that produce bad coordinates it’s the only real backstop, since you can’t clamp a position the way you clamp a number.
• Staged unlock, so “didn’t think of it” becomes “saw it happen, safely”. You don’t go from “software controls the machine” to “full force on a real part” in one step. You pass through stages, and each one bounds the consequences of whatever you haven’t found yet. Dry run: the machine doesn’t move; the software runs for real and logs every command it would have sent, so a wrong parameter costs a line in a log. Air cut: real motion, tool off, path raised above the work, so a wrong path draws a shape in the air. Scrap material: real force on a piece you’re happy to throw away. Then production. The bug you didn’t think of is still there at every stage. You never removed it. But it shows up at a stage where its consequence is bounded.

That’s what industrial reliability actually means. Not that the machine won’t have problems. It will. It means the problem shows up in the air above the work, with the tool switched off, and nobody gets hurt.

What vendors should say out loud

Here's something I think every vendor of anything that moves should be willing to say: we are not building toward "no bugs". There will be bugs, including some we'll never find. We're building so that when a bug fires, it gets caught, or bounded, or physically can't turn into harm. That can sound like an admission of weakness. I think it's the opposite. Someone who promises bug-free software is either naive or selling. Someone who shows you the stages, the two independent clamps and the box the arm can't leave has thought about the day things go wrong, which is the day that matters. "Bug-free" is a marketing claim; "consequence-bounded" is a design.

Written from a real force-controlled robotics integration, with all client, vendor and product details removed. The method is general: it applies anywhere a software failure has physical or irreversible consequences, from motion control to payments to deployments.

Where this comes from

None of this is my invention. It's established safety engineering, in plain words:

• “Testing shows the presence, not the absence, of bugs.” Edsger Dijkstra, 1969. This is where “you can’t test for what you didn’t foresee” comes from.
• Nancy Leveson, Engineering a Safer World (MIT Press), on why safety isn’t the same as reliability, and why you constrain the system instead of trying to perfect every part.
• IEC 61511, Layers of Protection Analysis: the industrial standard behind independent backstops (a safety layer must keep working even if the control system fails).