“Human-in-the-loop” became our favourite comfort phrase. Every AI conversation with a nervous stakeholder eventually reached it. Don’t worry, there’s a human in the loop. Nods, relief, next agenda item.
Then I started looking closely at what was actually happening in some of our loops, and the comfort evaporated.
In one review step, a person was approving AI-generated summaries at a pace that made real scrutiny impossible. The outputs were usually right, so approving them became routine, so scrutiny dropped further. Technically, a human was in the loop. Practically, the human was a rubber stamp with a pulse. And here’s the perverse part: the better the AI performed, the worse the oversight became, because sustained vigilance against rare errors is something humans are reliably bad at. Aviation and process safety have known this for decades. We were rediscovering it with fresh surprise.
The phrase itself was misleading us. It describes a location, in the loop, when what matters is a capability. So we rewrote our question. Not “is there a human in the loop?” but three harder ones. Can this person actually judge the output, do they have the expertise to know when it’s wrong? Do they have the time and information to exercise that judgement, or does the workflow quietly punish them for slowing down? And do they own the outcome, does their name sit on the result?
Judgement, capacity, accountability. Where any of the three is missing, the human isn’t oversight. They’re upholstery.
Some of our loops passed. Others we redesigned: sampling deep reviews rather than skimming everything, routing only the uncertain cases to humans, making the reviewer’s accountability explicit. In a few cases we concluded the honest answer was that the human added nothing, and the real control needed to sit elsewhere.
When someone tells you there’s a human in the loop, and someone will this week, ask what that human can catch, when they last caught something, and what happened next. The answer tells you whether you have a control or a comfort blanket.