The Loop is Broken: Five Structural Flaws Undermining Human Oversight of Generative AI
Introduction
Human in the loop (HITL) is the standard answer to a simple worry: what happens when an AI system gets it wrong? The reassurance usually sounds something like this: a person is reviewing the output, so mistakes will get caught before they cause harm. You'll find that reassurance in vendor marketing, in internal AI policies, and now in regulation itself.
This paper argues that, as commonly implemented for generative AI (GenAI) chatbots, that reassurance does not hold up. The problem is not that individual humans are careless, or that any one HITL policy is poorly written. The problem is structural. The way people actually review AI output runs into well-documented limits of human psychology, and generative AI in particular is well suited to work around those limits in ways that older, static AI systems were not.
I'll start by defining and scoping HITL so the argument applies consistently. From there, I'll walk through my five biggest concerns, each backed by research and real-world examples. Then I'll get into what actually can be done about it, and close with a note on how this analysis changes for agentic AI systems, which are outside the scope of the main argument.
What Is HITL?
Before making the case against how HITL is commonly relied on, I need to define what HITL is and set some boundaries around AI scope. It's a surprisingly large topic.
Scoping the AI
This paper is scoped to generative AI (GenAI) technologies: chatbots and similar systems built for back-and-forth communication with a user, where the goal is typically a written output that explains a topic, assists with a decision, or supports an iterative, co-editing style of work. This is distinct from agentic AI, which is its own subject with its own dynamics. I address it separately in the closing note on agentic systems, rather than folding it into the main argument here.
Scoping HITL
The HITL definitions I'm using below come from Lazaros et al., who define human in the loop, in its broadest sense, as the overarching design paradigm by which human input has operational impact on model development, deployment, supervision, or governance.
I'm scoping HITL more narrowly than that broad definition, narrowly enough to be useful, but in a way that should still be recognizable to most readers.
First, when is HITL applied? It can apply at several points in the AI lifecycle:
Inference time: when a user is actively using the AI system. This is the paper's focus.
Other stages: training, RLHF, data curation and labeling pipelines, and so on. These are out of scope for this paper.
Next, what type of HITL? There are five recognized types worth addressing:
In the loop. Humans have direct participation in every AI decision, typically providing authorization before the AI can proceed. In its strict sense, this is mostly reserved for agentic AI systems, so it's out of scope here.
On the loop. Humans monitor the AI's operation and intervene when necessary, the way an AI conducting automated trades or an AI driving a car might let a person take over. This is also an agentic-AI concept rather than a chatbot one, so it's out of scope in the technical sense.
Over the loop. The human defines objectives, constraints, and policy (essentially, the strategy), and the AI executes within those bounds. For example, a human might give an AI credit-scoring system the constraints for approving or denying loans and then let it run autonomously within them. This, too, is mostly an agentic-AI concept.
Under the loop. The human executes the final action based on the AI's output, and the AI provides guidance, decision support, or recommendations. This is the dominant use case for GenAI chatbots and is squarely in scope.
Along the loop. Parallel collaboration and co-creation on a task with the AI. This applies to agentic systems too, but it also applies heavily to GenAI chatbots, so it's in scope.
So, technically speaking, the HITL this paper addresses is Human Under the Loop and Human Along the Loop. I've titled the paper “Human in the Loop” anyway, since all five types are commonly grouped under that broader umbrella term.
With that scoping out of the way, here are the five problems I see with how HITL actually gets used for GenAI chatbots.
Loss of auditor independence. The person reviewing the AI's output is usually the same person who produced it with the AI's help, which quietly breaks the independence that review is supposed to provide.
AI personalization erodes independence further. Chatbots increasingly remember who you are and what you care about, which gives them more room to shape answers you're inclined to accept.
Sycophancy and persuasion bombing. GenAI doesn't just agree with you. It produces increasingly persuasive responses when challenged.
Automation bias and complacency. The more reliable an AI system seems, the less carefully humans tend to check its work, which is exactly backward from what you'd want.
Scapegoat in the loop. In a lot of real deployments, the human in the loop isn't there to catch errors. They're there to have someone to blame when errors happen.
Each of these gets its own section below, with the research and real-world examples behind it.
1. Loss of Auditor Independence
A common defense of human-in-the-loop practices is that the human user of an AI system will review its output and catch mistakes. Anyone familiar with IT audit practices (the kind governed by ISACA) or with audit practices in the financial industry should immediately see the problem: loss of auditor independence.
Users of AI systems may not think of themselves as “auditors,” but in a way, they are. They evaluate the output of their own AI system for accuracy before accepting or rejecting it. That may not sound like a problem on its face, but assessing your own work is a problem.
On January 8, 2003, a small propeller-driven aircraft carrying 21 passengers and crew crashed 35 seconds after takeoff, killing everyone aboard (Air Midwest Flight 5481). The plane pitched up, stalled, and descended uncontrollably almost immediately. The crash followed a botched maintenance job on the aircraft's elevator control system, and while any air crash has many contributing factors, one is directly relevant here: the duties of quality assurance inspector and mechanic were blurred.
The quality assurance inspector had provided the mechanic's on-the-job training, supervised the mechanic's work, and also acted as the independent inspector certifying that the repair was correct, all three at once.
This resulted in NTSB Safety Recommendation A-04-9, which prohibits inspectors from performing required inspection-item checks on any maintenance task for which the inspector had provided the on-the-job training to the mechanic who did the work. (The full accident report is available from the NTSB as well.)
This points to a real problem in human psychology: we cannot train & supervise someone doing work (or do the work ourselves) and also act as the independent inspector certifying its accuracy. I'm not a psychologist and I'm not qualified to speak to the underlying psychological mechanisms behind auditor independence, so I'll just note a couple of relevant findings from my research:
The proofreader's illusion. We fail to catch our own typos because the brain generalizes simple component parts, like turning letters into words and words into sentences, so it can focus on more complex tasks. When we re-read our own work, we miss typos because what's on the screen competes with the version that already exists in our heads (see Wired and this Oxford Reference entry on the effect). I'd guess this same mechanism helps explain the errors that show up when we lose auditor independence more generally. The inspector in the Air Midwest crash was, in effect, a writer caught in his own proofreader's illusion. He wrote the training and supervised the inspection, so when he reviewed the finished repair, his brain supplied the version of a job well done that already existed in his head, not the errors actually sitting in front of him.
Confirmation bias. Put simply, this is our tendency to use information that supports our existing beliefs and disregard information that doesn't (see the Catalogue of Bias entry). Confirmation bias isn't limited to arguments and essays. Even cases where a nurse or pharmacist accidentally administers the wrong medication can be attributed in part to confirmation bias: if the correct medication comes in packaging similar to the wrong medication, the brain may simply fail to register the label information that contradicts the belief that the correct drug is already in hand (see this case study).
“Saying that X with AI is ‘no different’ than X with a human is only true if you ignore everything that actually makes AI different.”
I posed this point to a group of information security professionals: reviewing a financial analysis your AI generated is not the same as reviewing one your co-worker wrote. Some people disagreed out loud, but I had the sense that most of the room was in silent disagreement.
The pushback I got was, how is it any different? Whether a person reviews a financial analysis generated by their AI or one written by and handed to them from a coworker, aren't they doing the same job? Reviewing a statement is reviewing a statement.
My answer is that it isn't the same at all. When a user provides financial details to a chatbot, answers its follow-up questions, asks for tweaks or explanations, and finally receives the output, how is that same person supposed to independently verify the resulting analysis with any real scrutiny? By contrast, if a coworker wrote the analysis (even with AI assistance), the reviewer is genuinely independent, assuming they didn't help produce the original analysis. From an independent-review standpoint, these two situations are entirely different, which is exactly why frameworks and legislation like the NTSB's, ISACA's, COSO, and SOX all include language on auditor independence or segregation of duties.
I often hear a version of the argument, “how is X any different with AI than with a human?” Sometimes it's taken as far as, “how is giving an AI agent credentials any different than giving a human credentials?”
In response, I'd like to propose an addition to the list of philosophical razors. Call it Oris's razor: “saying that X with AI is ‘no different’ than X with a human is only true if you ignore everything that actually makes AI different.” In other words, the claim that a task done with AI in the mix is no different than the same task done without AI is usually a mistaken assumption.
This razor holds even for seemingly simple assumptions, like the idea that validating AI output is no different from validating a fellow human's output. On that specific point, Randazzo et al.'s research suggests our normal, human-style validation techniques aren't well suited to validating AI outputs, because of the system’s persuasiveness and sycophancy. More on that in Section 3.
This concern about loss of auditor independence only gets worse once you factor in the next concern.
2. AI Personalization Erodes Independence
This is more of an a priori argument, but it's worth making: AI personalization can add further bias and deepen the loss-of-auditor-independence problem above.
Many AI chatbots have a memory feature that captures relevant data and context about the user, so the AI can give more tailored responses. This memory-building happens in the background, and users are often unaware of just how many personal details their AI chatbot has picked up. My own Claude and ChatGPT memory contains generic details, like the type of cybersecurity work I do and which certifications I hold, but it also knows my creative pursuits, how I prefer to learn, and my personal and professional interests. It can also cross-reference other chat history to build an even fuller picture.
From this information, an AI system could pick up on a user's personal biases and present erroneous information in a way that user is more likely to accept, rather than presenting the information at face value. To be clear, this is an a priori argument, not one based on personal observation. Still, going back to the earlier hypothetical of asking AI to conduct a financial analysis, it's worth taking seriously the possibility that the AI's analysis is shaped by everything it knows about its user.
It's important not to underestimate this effect. Glickman and Sharot's research on how human-AI feedback loops alter human perception found that users systematically underestimate how much the AI actually influences them.
A key finding from that research: participants reported feeling more influenced by an accurate algorithm than by a biased one, but in reality they couldn't tell the difference between the two. The degree to which participants became more biased when interacting with a biased algorithm was roughly equal to the degree to which they became more accurate when interacting with an accurate one, and participants had no sense of just how strong the biased algorithm's effect on them actually was. For our purposes, that suggests if your AI is biased, whether from what it knows about you or otherwise, you probably wouldn't notice.
3. AI Is Sycophantic: You Are Being Persuasion Bombed
“Sycophantic” has become common parlance thanks to AI. In an AI context, it's easy to define as the AI constantly agreeing with everything you say. The fear is that an agreeable AI that never pushes back will reinforce confirmation bias and discourage critical thinking.
But, there is more to sycophancy than just agreement.
A working paper by Randazzo et al. (still in draft form, but useful for this discussion) clarifies that sycophancy isn't just static agreement. In generative AI, it takes the form of what the authors call persuasion bombing.
Prior HITL research looked at AI generally, not GenAI specifically. The distinction the researchers draw is that older AI systems produce static forecasts, while GenAI generates novel outputs and can modify those outputs in real time based on human input.
That's a critical difference. With a static AI forecast, existing research describes “engaged AI interrogation,” where humans question and validate the AI's output. This works well because the output doesn't change: when professionals validate it, they stay separate from the decision-making process (the independence requirement from Section 1) and validate externally, through independent research, consulting colleagues, or referencing outside sources such as books or web searches.
With generative AI, that validation happens inside the GenAI platform itself instead. There's no more independent research, just a continued discussion with the same chatbot.
That's what leads to persuasion bombing: GenAI producing increasingly persuasive responses to its early or preliminary output the more the user pushes back. The AI's ability to modify its output in real time based on human input is what enables this, and it's what makes human-in-the-loop review meaningfully harder for GenAI systems than it was for older, static AI systems.
The authors describe the AI's tactics as “insidious persuasive tactics.” I thought that was overkill at first, but after reading the paper, I'm convinced “insidious” is the right word.
That's because the tactics aren't obvious. They're almost subliminal, which shouldn't be surprising: if the persuasion were obvious, you'd notice it, and it would fail.
According to the research, users tend to validate GenAI output in three general ways:
Fact-checking: asking the GenAI to re-check its own analysis (“did you consider everything?”). Note that this isn't fact-checking via an outside source like a search engine. It's asking the AI to check itself.
Exposing: pointing out a logical or factual flaw or contradiction directly to the AI (“does this really make sense, given that…?”).
Pushing back: disagreeing with the AI and asking it to reconsider. This is distinct from fact-checking, since here the user is asking for a different position rather than a double-check.
The problem is that none of these three tactics involve validating the AI's output outside of the AI itself.
Watching user interactions in a test setting, the researchers saw the AI use a wide range of persuasion tactics, which they mapped back to Aristotle's classic modes of persuasion: Ethos, Pathos, and Logos.
Ethos: appealing to the character and credibility of the speaker
The AI applied Ethos tactics such as apologizing (which sounds innocent, but psychologically builds trust and rapport), using corrective language like “upon closer examination,” using ‘effort transparency’ language like “after careful evaluation” (in what sense can an AI even be “careful”?), and hedging its bets by offering multiple answers.
None of that may seem important. So what if the AI apologized or said it was being thorough? But it matters a lot. It's how the AI bolsters its perceived credibility and builds rapport, making you more receptive to its answers, not unlike a con man building trust before making the ask.
Logos: appealing to the logical validity of the argument
The researchers found GenAI framing its responses with data, comparisons, and structured reasoning. That sounds like a good thing. If the AI is giving raw data and structured reasoning, isn't that HITL working as intended?
Not quite. GenAI wasn't using data, comparisons, and structured reasoning simply to convey facts. It was using them as persuasion tactics to convince users of a position, even when the underlying analysis was flawed.
In one case, the AI agreed with users and then immediately offered a counterpoint, a fairly obvious tactic. More subtle was its use of data: in one example, when suggesting an upward trend in a market segment, the AI cited specific figures, going from “45.20%” to “52.10%.”
That might look like simple accuracy and transparency rather than persuasion, but the problem isn't the data itself. It's how and when the numbers get deployed. This produces biases like the “appearance of precision” and can trigger the Cognitive Miser effect, the well-documented tendency for humans to take mental shortcuts and expend as little mental effort as possible on a problem. Consider the classic example: a bat and ball together cost $1.10, and the bat costs $1 more than the ball. How much does the ball cost? The easy, wrong answer is $0.10. The correct answer is $0.05 (see Yale Insights).
The point is that just seeing or hearing statistics can persuade our brains. Dropping a number or stat at the right moment can persuade people even when the stat is wrong or doesn't actually support the argument. Just look at politicians and debaters.
One might object that this is a human psychology flaw, not an AI flaw. And it's true, people fall victim to Logos-laden arguments and the Cognitive Miser effect outside of AI too. That's exactly what the politicians-and-debaters example shows.
To that, I'd again point to Oris's razor (look, at least one person is using it). With standard AI producing static output, users were found to validate that output externally, consulting the internet, books, or coworkers. With GenAI, users see the impressive-looking numbers and conclude that the increase from 45% to 52% must support the AI's claim, without checking whether those numbers are actually relevant or actually back up the claim being made. The mere presence of a number is often enough to persuade.
Pathos: appealing to the emotional responses of the audience
The most obvious mode of persuasion is emotional appeal. GenAI was found to affirm users' existing beliefs (what most people default to calling “sycophancy”), use energetic responses, mirror the user's own language, and make bold future aspirations.
What I found more novel in this research wasn't the mere presence of Ethos, Pathos, and Logos tactics on their own (that alone wouldn't qualify as insidious to me), but how the AI behaved when users pushed back. In one example, an AI's initial output contained two persuasion tactics. After the user pushed back, its next output contained five.
Now overlay this “power persuader” behavior, as the authors call it, with the personalization concern from Section 2. You end up with a power persuader that also has a deep understanding of exactly what makes its specific audience, you, tick. Add to that the fact that virtually no one seems to be fact-checking their AI output outside the AI platform itself (even professionals whose job is deploying AI don't always bother, as this PCMag report on a hallucination-filled AI report published by KPMG shows), and the scale of the problem for human-in-the-loop validation becomes clear.
One footnote worth adding: even if the AI isn't persuading you, or you find yourself somehow impervious to it, the authors cite other studies showing that when an AI model produces an unexpected or surprising output that contradicts the human decision-maker's intuition, the human is more likely to reject the AI's recommendation outright. Maybe some decisions really should just be left to humans.
Sources for this section: the concept of persuasion bombing throughout is drawn from Randazzo et al.'s working paper.
4. Automation Bias and Complacency
To introduce automation bias and complacency, it's worth going back to 1993, to an experiment that tested how increasing automation in an aircraft cockpit led to what the researchers called “automation-induced complacency.” Complacency, in this study, was defined as an operator failing to detect a failure in the automated control of a system-monitoring task.
In the experiment, subjects monitored a simulated panel of cockpit instruments and aircraft conditions: fuel management, communications, engine temperature, and more. Constant monitoring of several systems at once is difficult, so automation was introduced to sound a warning alarm when an issue arose. But the researchers made that automation system intentionally flawed. For some users, the system was fairly accurate (though still below 100%) but consistent. For others, the system's accuracy changed every ten minutes, alternating between highly accurate and inaccurate (inconsistent). The finding was that when a system is flawed but consistent, users become complacent and start missing automation errors.
This happens because automation shifts a human's role from active control to system monitoring, and a monitoring role is fundamentally different from an active control role. Not all humans are trained to monitor, audit, or evaluate systems well.
The study suggests that when automation is very reliable, upwards of 99%, that reliability can actually set up a catastrophic automation failure. Highly reliable automation fails less often, but when it does fail (a matter of when, not if), humans are unlikely to catch it.
As the researchers put it, automated monitoring does not necessarily provide true redundancy to human monitoring, because the presence of automation itself affects the efficiency of human monitoring. For our purposes, human in the loop is not as accurate as we tend to give it credit for and should not automatically be assumed to be an independent, additive, or effective control. A good HITL review can improve results, and there are suggestions at the end of this paper on how to do that, but an organization needs to be honest with itself on the robustness of their HITL controls and how much improvement is actually gained through the use of human review.
But what do we find in these modern, AI-driven times? Romeo and Conti's literature review agrees with the 1993 cockpit-automation study. They cite research reinforcing that higher automation accuracy increases user trust, which in turn increases automation bias. (I personally think of automation bias as arising from psychological phenomena the user may not be aware of, and automation complacency as more of a material effect driven by tiredness, boredom, or frustration, though the literature is divided on this distinction).
They also found that explainable AI (XAI) and AI transparency are often insufficient to mitigate automation bias. Highly explainable outputs can be overly technical and cognitively demanding, triggering the same Cognitive Miser effect from Section 3, which leads to automation bias. Alternatively, XAI can produce oversimplified explanations that inadvertently increase trust in the system and lead to automation bias by a different route.
Similar findings show up in cybersecurity specifically. One study found that participants who placed high trust in an anti-virus system were far less capable of distinguishing safe emails from phishing emails, and became complacent about security generally. Another found that people operating under higher levels of automation were less likely to detect security threats at all.
Of course, this doesn't mean we should abandon automation. The same authors cite a paper concluding that Target's 2013 data breach arose from disuse and lack of trust in security automation. A tool that could have automatically detected and deleted malicious files had simply been left disabled.
Applied to AI and human in the loop, the takeaway is this: as AI becomes more and more accurate (but not 100% accurate), automation bias and complacency take over, and HITL will not be as accurate a control as we tend to assume. This may happen for psychological reasons users aren't even aware of, or through plain complacency, as seen with AI coding tools like Claude Code. In one illustrative example, a user created a fictional keyboard with three buttons, “allow once,” “always allow,” and “reject,” with the “always allow” button rendered the biggest. It's a pointed joke about a real phenomenon: users who are frustrated, short on time, or otherwise don't feel like reviewing the code an AI agent is about to run end up putting the AI in “always allow” mode, cutting out human checks altogether.
5. Scapegoat in the Loop
The last problem I'll cover here, though not the last problem with human in the loop generally, is what researchers call “scapegoat in the loop.” The concept captures situations where humans are nominally included in an AI system primarily to absorb responsibility, rather than to provide any meaningful oversight.
Even when human in the loop is added to an organization's policies and training with the best of intentions, the actual implementation often ends up functioning as a scapegoat in the loop. HITL serves as legal or regulatory cover, but doesn't do much to actually improve the accuracy of the system.
This gets worse once you consider where the blame lands when the AI fails. The expectation is that a human will catch the error. If they don't, the human is blamed. This puts humans in what researchers call a “moral crumple zone,” one that protects the integrity of the machine at the price of placing disproportionate, or even unjust, blame on the nearest human.
This becomes even more worrisome as legislation increasingly enforces human oversight of AI systems, as seen in Article 14 of the EU AI Act. Granted, human oversight requirements like this aren't only meant to improve a system's accuracy, they're also meant to protect rights, allow escalation, and stop an unsafe action before it causes harm. I don't disagree with the sentiment behind that kind of legislation, it is truly important, but I disagree about the degree to which it will actually help. Based on everything above, human oversight is not strong enough to reliably catch errors and puts human reviewers helplessly in the moral crumple zone.
A related concern is that organizations may forgo other AI controls on the assumption that, if anything goes wrong, a human in the loop will catch it.
In summary, telling people to “double check the AI's output” or to “maintain your professional judgment” simply does not work reliably enough to justify placing blame on humans, or to justify forgoing other AI controls.
Sources: this section draws on Lazaros et al. and related research on the scapegoat-in-the-loop concept.
What Do We Do About It?
Having laid out the case for the flaws and limitations of human in the loop, the natural next question is: what do we actually do about it?
I'll start with a proposed solution from the 1993 cockpit-automation paper, because I think it's a genuinely terrible solution. It's amusing to consider, but not one I'd recommend. That study found that when automation is consistently reliable (but not 100% reliable), humans become complacent, but when a system is unreliable and makes occasional errors, humans are more likely to stay skeptical, avoid complacency, and catch errors, possibly even increasing the system's overall accuracy.
One idea the paper floats is to deliberately introduce artificial failures into a live automation system, just to keep the operator on their toes and require a response. I don't know about you, but I'd really rather not have an airplane cockpit tell pilots there's an engine fire when there isn't, just to keep them sharp.
So, setting aside the idea of intentionally degrading AI to fight automation bias, what actually works?
A. Timing
In most cases, a human receives the AI's output and then has to review it or catch errors. This leaves the human exposed to all the problems described above. One finding, though, points to a useful exception: human judgment tends to be more accurate when the human's own verdict is formed before receiving an erroneous AI recommendation, rather than after.
In other words, if you force the human to make a decision first and only then bring in the AI for assistance, the human is more likely to reach the correct verdict, even when the AI itself is wrong.
This won't work everywhere, but it's reasonable in situations like hiring, where two candidates' resumes are being evaluated. Have the hiring manager pick who they think is the best candidate first and justify their reasoning, and only then bring in AI for decision assistance.
B. Decouple Tasks
Have someone other than the original user review the AI's output (Bias in the Loop) and validate the results with non-AI systems. This addresses most concerns raised above. It restores independence, it means someone else can check whether the output was biased toward the original user's preferences, the reviewer isn't the one being persuasion-bombed in real time, it meaningfully reduces automation bias and confirmation bias for that reviewer, though an independent reviewer can still place too much trust in a polished AI answer, and it keeps that person from being a scapegoat in the loop.
This isn't feasible in every situation, but where it is, it's a strong defense.
C. Scope AI to Correct Roles and Domain Experts
This won't address many of the concerns above, persuasion bombing especially, but it's still important: an AI scoped to a specific domain (a contract-review AI bot, for example) should be limited to users who already have knowledge in that domain, like your legal team. The fact that an AI system reviews contracts is not an excuse to let users without legal training review NDAs on the assumption that the AI will close the knowledge gap. The AI may misinterpret something your legal department would catch, and you wouldn't.
An illustrative example: credit card scammers who inadvertently leaked the details of 345,000 stolen cards. The scammers used Cursor, a legitimate AI coding agent, to build a server that tests if the stolen cards are still active. What Cursor created was a web server with an unauthenticated, open web directory exposed to anyone. This isn't necessarily Cursor's fault. As Steve Gibson put it on his Security Now! podcast, “I think it's a great example of the danger of using AI without being a domain expert… I've no doubt that the Cursor AI would have easily provided instructions for the authentication that was needed if it had been asked to. But apparently the bad guys never thought to ask. Somebody who wasn't really up to speed on web-based application security could easily fail to anticipate all the various ways others might access and penetrate their system. So, expecting AI to produce secure solutions by default is probably a fool's errand.” (Show notes.) Anthropic has made a related point elsewhere, addressed in the note on agentic systems below.
D. Test the AI's Accuracy and Accept Failures (If Possible)
If an AI system is 98% accurate, adding a human in the loop will likely not push it to 99%, based on everything covered above. One way to address this directly is to rigorously test your AI systems, compute their accuracy on the required task, and confirm that accuracy is higher than what a human would achieve on their own.
Take a hypothetical: an AI system meant to help pharmacists dispense medications. Pharmacists are estimated to make about 5.7 dispensing errors per 10,000 prescriptions.
If we wanted to outsource part of this task to AI, we could test the system's accuracy directly. If that AI makes 8 errors per 10,000, it's not realistic to expect the pharmacist's HITL review to reliably catch the 2.3 or more additional errors needed just to bring the combined system back to human-level accuracy.
Now say you find a different AI system that makes only 3 errors per 10,000, better than the human average. In that case, even when the AI does err, it's still erring less often than a human would on their own. Therefore, you could deploy that system with the assumption that HITL is going to fail and accept that when an uncaught error does occur, you are still performing better than in a human only system anyway. You could still maintain a HITL policy for legal purposes, but you'd write that policy knowing the human likely would not catch the error, so you shouldn't place the pharmacist in that moral crumple zone and then punish them for the inevitable outcome of well-documented psychological limits on how effective HITL actually is.
E. Deploy and Maintain Controls
As covered above, human in the loop does not work well enough that we can afford to sacrifice or scale back other controls.
Deploying AI responsibly is not as simple as turning a chatbot loose on your enterprise. Plenty of architectural and data decisions affect an AI system's accuracy. You could run additional AI systems, like multi-agent or sub-agent setups, or a sentinel, to check outputs for errors. You could also fine-tune the model, ground its data with RAG, or set guardrails that shape the AI's behavior and accuracy directly.
A Note on Agentic AI Systems
As stated at the outset, this paper is scoped to generative AI chatbots, not agentic systems like Claude Code. I've kept those separate because HITL works quite differently for agents.
Claude Code, for example, will stop and show the user exactly what command the agent is about to run (an approval gate). If Claude says it's about to run an “rm -rf” command, most users should recognize what that means and decline it, unless that was actually their intent. Agentic systems create a lot more room for approval gates. An agent can require out-of-band approval through in-app prompts (as in Claude Code), through workflows in automation platforms like n8n, or via tickets, emails, or text messages, and these approval requests can be supplemented with concrete details, like the exact command that will run. This is a fundamentally different situation from the AI chatbots discussed throughout this paper, where users have a back-and-forth discussion and the output is generally text meant to persuade, educate, or share an opinion, rather than code or commands being executed directly. (Anthropic has written about this.)
Most of the mitigations I proposed don't translate to agentic systems. You can't approve AI-written code before asking it to write the code, and decoupling roles doesn't work the same way either. The domain-expertise idea from mitigation C does apply. Anthropic has noted that experienced users auto-approve roughly twice as often as new users, but experienced users also have the knowledge to, and more frequently do, interrupt the agent mid-execution when something looks like it's going sideways. In most GenAI chatbot use cases, I don't see this as a viable substitute. HR is not going to watch an AI decide between two job candidates in real time and catch it going off the rails mid-execution, and likely couldn't even if they tried.
Another difference with agentic systems, one that would justify an entire paper of its own, is the availability of additional guardrails and controls like classifiers and sandboxes. In response to approval fatigue, Anthropic ran an agent in a sandbox where all reads were allowed, all writes were allowed inside the workspace, and network connectivity was denied entirely. Inside that sandbox, HITL interruption was mostly unnecessary, and the approach cut permission prompts by 84%, significantly reducing approval fatigue. This kind of control doesn't have an equivalent for most chatbot deployments.
Close
None of this is an argument that human oversight of AI is worthless, or that HITL should be abandoned. It's an argument that the version of HITL most organizations actually implement, a single user reviewing the output of the same AI system they just used to produce it, rests on assumptions about human attention, independence, and resistance to persuasion that don't hold up under examination.
Loss of auditor independence, AI personalization, persuasion bombing, automation bias and complacency, and the scapegoat-in-the-loop dynamic all point in the same direction: a human “in the loop” is not, by itself, a control you can rely on for accuracy. Treating it as one lets organizations under-invest in the controls that actually would help: decoupled review, domain-scoped access, tested and measured AI accuracy, and real architectural guardrails.
For agentic AI systems specifically, some of these dynamics play out differently, and approval gates paired with sandboxing offer options that don't exist for chatbot-style GenAI. But that's a distinct discussion, and one I've intentionally left for another time.
The goal here isn't to argue against oversight. It's to argue for being honest about what “a human is reviewing it” actually buys you, so that the controls organizations rely on are the ones that actually work, not just the ones that are easiest to write into a policy document.
AI Transparency Statement
I believe it's important to be transparent when using AI. Readers deserve to know how and when AI was used to produce a piece of work, so here's how I used it in my research and writing.
In line with mitigation A, timing, I started without any help from AI. I found articles and papers on my own and worked out the themes independently before bringing AI into the process.
Once I had that foundation, I uploaded the papers I'd found to Claude and had it search online for additional related articles and concepts. With AI's help tracking down more sources, I ended up with 10 primary sources (papers directly related to HITL) and 13 supplementary sources (papers, articles, and other material supporting specific arguments and tangents).
Before using AI at all, I'd already put together a list of common themes and concerns about HITL. I later used Claude and NotebookLM to identify themes across all the sources independently, and only then compared their output to my own list. Claude and NotebookLM were also useful throughout research and writing for organizing sources, surfacing evidence and quotes, and answering questions about what a given source actually said.
Every time the AI pulled a quote, finding, or statistic from one of my sources, I went back and read that section myself to confirm the source hadn't been misquoted and that the finding was actually relevant to my point. The AI never misquoted a source. On two occasions, though, it presented a finding as supportive of my argument that, once I read the source in full context, I decided didn't actually hold up. I left those out.
I wrote the paper itself: the arguments, conclusions, and examples are mine. After writing, I used Claude as a style editor to clarify my language and improve readability. It didn't change any arguments, reorganize the paper, or invent or suggest any examples or anecdotes. I compared my original draft to the AI-edited version line by line to confirm that only tone, style, and readability had changed, and that no new claims or shifts in meaning had crept in.
Sources
This is not a published work, so I am knowingly choosing not to follow strict APA citation requirements. All sources used in this work are linked throughout and listed below.
Agudo, U., Liberal, K.G., Arrese, M. et al. The impact of AI errors in a human-in-the-loop process. Cogn. Research 9, 1 (2024). https://doi.org/10.1186/s41235-023-00529-3
Anthropic, (2026, May 25). How we contain Claude across products. https://www.anthropic.com/engineering/how-we-contain-claude
Beck, Jacob & Eckman, Stephanie & Kern, Christoph & Kreuter, Frauke. (2025). Bias in the Loop: How Humans Evaluate AI-Generated Suggestions. 10.48550/arXiv.2509.08514. https://www.researchgate.net/publication/395403018_Bias_in_the_Loop_How_Humans_Evaluate_AI-Generated_Suggestions
Catalogue of Bias Collaboration, Spencer EA, Heneghan C. (2018). Confirmation Bias. In: Catalogue of Bias 2018. https://catalogofbias.org/biases/confirmation-bias/
Colombo, E. (2026, January 16. Version 1). The Sentinel System: A Global Architecture For Ai Governance And Safety. https://client.prod.orp.cambridge.org/engage/coe/article-details/6960efa75c91413e6ef9ffa4
EU AI Act, (retrieved 2026, July 15). Article 14: Human Oversight. https://artificialintelligenceact.eu/article/14/
Frederick, S. (2009, November 27). Why smart people do dumb things. https://som.yale.edu/news/2009/11/why-smart-people-do-dumb-things
Glickman, M., Sharot, T. How human–AI feedback loops alter human perceptual, emotional and social judgements. Nat Hum Behav 9, 345–359 (2025). https://doi.org/10.1038/s41562-024-02077-2
Grissinger M. (2012). 'Inattentional blindness': what captures your attention?. P & T : a peer-reviewed journal for formulary management, 37(10), 542–555. https://pmc.ncbi.nlm.nih.gov/articles/PMC3474444/
Lazaros, K., Vrahatis, A. G., & Kotsiantis, S. (2026). Human-in-the-Loop Artificial Intelligence: A Systematic Review of Concepts, Methods, and Applications. Entropy, 28(4), 377. https://doi.org/10.3390/e28040377
McCurdy, W. (2026, June 15). KPMG Allegedly Published AI Report Filled With Hallucinations. https://www.pcmag.com/news/kpmg-allegedly-published-ai-report-filled-with-hallucinations
NTSB, (2004, March 5). Safety Recommendation 7545B. https://www.ntsb.gov/safety/safety-recs/recletters/A04_04_24.pdf
NTSB, (2003, January 8). Aircraft Accident Report Loss of Pitch Control During Takeoff Air Midwest Flight 5481. https://www.ntsb.gov/investigations/AccidentReports/Reports/AAR0401.pdf
Oxford Reference, (n.d.). Proof-reader’s illusion. https://www.oxfordreference.com/display/10.1093/oi/authority.20110803100349527
Parasuraman, Raja & Molloy, Robert & Singh, Indramani. (1993). Performance Consequences of Automation Induced Complacency. International Journal of Aviation Psychology. 3. 10.1207/s15327108ijap0301_1. https://www.researchgate.net/publication/23595844_Performance_Consequences_of_Automation_Induced_Complacency
Randazzo, Steven and Joshi, Akshita and Kellogg, Katherine and Lifshitz-Assaf, Hila and Dell'Acqua, Fabrizio and Lakhani, Karim R., GenAI as a Power Persuader: How Professionals Get Persuasion Bombed When They Attempt to Validate LLMs (July 01, 2025). Available at SSRN: https://ssrn.com/abstract=5678644 or http://dx.doi.org/10.2139/ssrn.5678644 or https://www.hbs.edu/ris/Publication%20Files/26-021_8db29bd1-04ed-4e98-86de-d286721afc7a.pdf
Ranisch R. (2024). Scapegoat-in-the-Loop? Human Control over Medical AI and the (Mis)Attribution of Responsibility. The American journal of bioethics : AJOB, 24(9), 116–117. https://doi.org/10.1080/15265161.2024.2377137
Reddit (2026, March). Just picked up a new keyboard - can't wait to write a bunch of code with it. https://www.reddit.com/r/Anthropic/comments/1rrrmug/just_picked_up_a_new_keyboard_cant_wait_to_write/
Romeo, G., Conti, D. Exploring automation bias in human–AI collaboration: a review and implications for explainable AI. AI & Soc 41, 259–278 (2026). https://doi.org/10.1007/s00146-025-02422-7
Security Now! Episode 1077 transcript (2026, May 5). A Browser AI API? https://www.grc.com/sn/sn-1077.htm
Security Now! Episode 1077 show notes (2026, May 5). A Browser AI API? https://www.grc.com/sn/sn-1077-notes.pdf
Stockton, N. (2014, August 12). What's Up With That: Why It's So Hard to Catch Your Own Typos. https://www.wired.com/2014/08/wuwt-typos/
The Cochran Firm (accessed July 14, 2026). Pharmacy Errors. https://www.cochranfirm.com/new-orleans/pharmacy-errors/
Tilbury, Jack & Flowerday, Stephen. (2024). Automation Bias and Complacency in Security Operation Centers. Computers. 13. 165. 10.3390/computers13070165. https://www.researchgate.net/publication/381973037_Automation_Bias_and_Complacency_in_Security_Operation_Centers