The last human decision

A lot of AI products now have a final step where a person approves what the system suggests. Accept this reply. Approve this candidate shortlist. Confirm this diagnosis. The design is meant to keep a human in charge. In practice, I think many of these approvals are closer to a signature on a form nobody reads.

As recommendations get better, this problem gets worse, not better. When the system is right almost every time, checking its work starts to feel like a waste of effort. So people stop checking, and the human in the loop becomes a rubber stamp. The approval step is still there. The decision has quietly moved somewhere else.

Automation bias is a design problem

Researchers have studied this for decades, starting with pilots and air traffic control. Kathleen Mosier, Linda Skitka and their colleagues gave it a name: automation bias, the tendency to use an automated suggestion as a shortcut instead of looking at the evidence yourself. They described two kinds of mistake it produces. Commission errors are when you follow the system’s advice even though other information says it’s wrong. Omission errors are when you miss a problem because the system didn’t flag it.

The second kind is the one I think product teams underestimate. In one of Skitka’s experiments, people who did a monitoring task with a very reliable, but not perfect, automated aid did worse than people who had no aid at all, largely because they missed events the aid didn’t prompt them about. The aid didn’t just fail to help. It changed where their attention went.

It isn’t laziness, either. Raja Parasuraman and Dietrich Manzey reviewed the research and found automation bias in experts as well as novices, and found that simple practice doesn’t make it go away. Their explanation is about attention. When you are busy, attention goes to whatever seems to need it most, and a system that is usually right seems to need it least. A systematic review of clinical decision support by Kate Goddard and colleagues found the same pattern in healthcare: heavier workload, more complex tasks and time pressure all made over-reliance more likely.

What I find important is that the interface shapes how strong it is. The same review found that design factors mattered, including where the advice sat on the screen and whether the system offered information or an outright recommendation. If the recommendation is shown first, in bold, with a pre-selected button, the person’s job quietly shifts from deciding to objecting. Objecting is harder. You need a reason, and you’re going against something that is usually right.

Why better systems make it harder

Lisanne Bainbridge pointed out the core problem back in 1983, in a short paper called “Ironies of automation”. The more we automate, the more we ask people to do the one thing humans are worst at: watch a system that rarely fails, then step in, at speed, when it does. And because the person rarely does the task themselves, the skill they would need to step in slowly fades.

There is a second mechanism that I think matters even more for AI products. John Lee and Katrina See described trust in automation as something that should be calibrated: high where the system is reliable, low where it isn’t. But calibration needs feedback. You learn where a system is weak by seeing it fail. If it is right 98 times out of 100, most people will never notice the two misses, so their trust settles on “always”. A very good system gives you fewer chances to learn where it is bad.

Zana Buçinca, Maja Malaya and Krzysztof Gajos make a related point. People rarely think hard about each individual recommendation. Instead they build a general rule of thumb about when to go along with the AI, and then apply it without much thought. That is efficient, and it is exactly what makes oversight fail on the case that matters.

The opposite failure exists too, and it’s worth holding both in mind. Berkeley Dietvorst, Joseph Simmons and Cade Massey found that people lose confidence in an algorithm faster than in a human after seeing it make the same mistake, and then avoid it even when it outperforms them. They called this algorithm aversion. So the goal isn’t simply “make people trust the system less”. Both rubber-stamping and rejecting a good system are calibration failures. The design problem is helping people trust it the right amount, in the right places.

That turns out to be hard even when people try. In a study of risk assessments in pretrial release and lending decisions, Ben Green and Yiling Chen found that participants couldn’t reliably judge whether their own predictions or the model’s were more accurate, and didn’t adjust how much they relied on it to match how well it performed. Changing how the model was presented helped in some ways, but not in these.

Which decisions should stay human?

I don’t think the answer is “all of them”. Most decisions are fine to automate, and pretending otherwise just adds friction. Asking a person to approve something they can’t meaningfully assess doesn’t make it safer. It just moves the blame. But I’ve found it useful to ask four questions about any decision a product makes for people.

• Can it be undone? Sorting your inbox can be reversed. Rejecting a job applicant usually can’t, at least not in a way that helps them.
• Who carries the consequence? If the person affected isn’t the person approving, the stakes of a careless click are higher, and the person approving has less reason to slow down.
• Does it depend on values, not just facts? Choosing the fastest route is a fact question. Choosing what to say to a grieving friend is a values question, and the “best” answer depends on who you are.
• Would the person want to be able to say “I chose this”? Some decisions matter partly because we make them. A wedding toast written by a tool might be better written, and still be worse.

If a decision scores low on all four, I think the honest design is to automate it and make it easy to undo. If it scores high on any of them, the human step has to be real. The worst outcome is the one in between: a decision that looks human on paper but isn’t in practice.

Designing a decision that’s really human

If a decision should stay human, the interface has to make human judgement possible, not just technically allowed. Some patterns that help, and what the research says about each:

Ask before you suggest. Asking the person for their own view first, then showing the system’s view, keeps their judgement independent. Buçinca and her colleagues tested designs like this, which they call cognitive forcing functions, and found they reduced over-reliance more than simply adding explanations did. But there was a catch. People liked the most effective designs the least, and the designs helped people who enjoy effortful thinking more than those who don’t. So this is a pattern to reserve for decisions that deserve it, not one to apply everywhere. It’s slower, and that’s the point, but slowness has to be spent where it matters.

Show the reasons, but don’t expect reasons alone to fix it. This is where I’ve changed my mind. I used to assume that explaining a recommendation would help people catch its mistakes. Gagan Bansal and colleagues tested exactly that, and found that explanations made people more likely to accept the AI’s recommendation whether it was right or wrong. An explanation can work as a signal of competence rather than as evidence. What seems to help more is showing how sure the system is, case by case, and showing information the person can check, rather than a verdict to agree with. Goddard’s review points the same way. A reason only helps if the person can actually test it.

Make disagreeing as easy as agreeing. If “Accept” is one click and “Change” is three screens, you’ve already made the choice for them. The same goes for visual weight, default focus, and what happens when someone presses Enter.

Limit the volume. One person approving 400 suggestions an hour isn’t reviewing them. Workload is one of the most consistent factors in this research. If a workflow needs that many decisions, either automate it honestly or change the workload.

Make the person feel answerable. In Mosier and Skitka’s cockpit study, telling pilots they would be held accountable didn’t change much. But pilots who said they personally felt responsible for how they used the automation were more likely to cross-check it and made fewer errors. That was a correlation found after the fact, so I hold it loosely. Still, it suggests that a decision should feel owned. Showing who decided, and asking for a short reason on the decisions that matter most, does more than a legal disclaimer.

Measure the oversight, not just the model. Teams track model accuracy closely and almost never track whether reviewers are adding anything. If people override the system almost never, and their overrides aren’t more often right than wrong, the human step isn’t doing its job, however good the approval screen looks.

A human in the loop only matters if the loop gives them room to think.

Regulation is catching up, slowly

The EU’s AI Act asks that high-risk systems be designed so people can effectively oversee them. Article 14 even names the problem: the people overseeing a system should be able to “remain aware of the possible tendency of automatically relying or over-relying on the output”, and be able to disregard, override or reverse it. That’s a useful principle. But a rule can’t specify what “effective” looks like on a screen.

Ben Green makes a sharper point. After reviewing 41 policies that require human oversight of government algorithms, he argues that they rest on an untested assumption, that people can actually do this kind of oversight, and that the evidence says they often can’t. Worse, a human sign-off can make a flawed system look legitimate and let everyone avoid responsibility for its mistakes. His proposal is that organisations should have to show evidence that their form of oversight works before they rely on it. I think that’s right, and it turns oversight into a design brief. Showing that the human step actually works is design work, and it will be decided one approval flow at a time.

Speed isn’t the only value

Automation is usually justified by speed and accuracy, and both matter. But some decisions are also about responsibility, dignity and meaning. A doctor who talks through a diagnosis, a manager who explains why someone didn’t get the job, a person who chooses their own words: these are slower, and they’re part of what makes the decision legitimate.

As designers, we choose where the human sits in the process and how much space they get there. I’d rather make that choice carefully now than discover later that we designed people out of decisions they would have wanted to keep.

References

12

1. Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. doi.org/10.1016/0005-1098(83)90046-8
2. Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T. & Weld, D. (2021). Does the whole exceed its parts? The effect of AI explanations on complementary team performance. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 1–16. doi.org/10.1145/3411764.3445717
3. Buçinca, Z., Malaya, M. B. & Gajos, K. Z. (2021). To trust or to think: Cognitive forcing functions can reduce overreliance on AI … Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1), 1–21. doi.org/10.1145/3449287
4. Dietvorst, B. J., Simmons, J. P. & Massey, C. (2015). Algorithm aversion: People erroneously avoid algorithms after seeing them err. Journal of Experimental Psychology: General, 144(1), 114–126. doi.org/10.1037/xge0000033
5. Goddard, K., Roudsari, A. & Wyatt, J. C. (2012). Automation bias: A systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 19(1), 121–127. doi.org/10.1136/amiajnl-2011-000089
6. Green, B. (2022). The flaws of policies requiring human oversight of government algorithms. Computer Law & Security Review, 45, 105681. doi.org/10.1016/j.clsr.2022.105681
7. Green, B. & Chen, Y. (2019). The principles and limits of algorithm-in-the-loop decision making. Proceedings of the ACM on Human-Computer Interaction, 3(CSCW), 1–24. doi.org/10.1145/3359152
8. Lee, J. D. & See, K. A. (2004). Trust in automation: Designing for appropriate reliance. Human Factors, 46(1), 50–80. doi.org/10.1518/hfes.46.1.50_30392
9. Mosier, K. L., Skitka, L. J., Heers, S. & Burdick, M. (1998). Automation bias: Decision making and performance in high-tech cockpits. The International Journal of Aviation Psychology, 8(1), 47–63. doi.org/10.1207/s15327108ijap0801_3
10. Parasuraman, R. & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. doi.org/10.1177/0018720810376055
11. Skitka, L. J., Mosier, K. L. & Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5), 991–1006. doi.org/10.1006/ijhc.1999.0252
12. Regulation (EU) 2024/1689 of the European Parliament and of the Council (Artificial Intelligence Act), Article 14: Human oversight. Official Journal of the European Union. eur-lex.europa.eu/eli/reg/2024/1689/oj