What happens when AI starts making decisions for us?

For most of the last few years, AI products have mostly talked. You ask a question, it gives an answer, and you decide what to do with it. If the answer is wrong, you can ignore it. Nothing has happened yet.

That is changing quickly. Assistants now book the meeting, send the email, file the expense, merge the code and pay the invoice. When a system that answers gets something wrong, you’ve wasted a minute. When a system that acts gets something wrong, something has already happened in the world, often to someone other than you.

In The last human decision I wrote about approval screens that quietly turn people into rubber stamps. This post is about the step before that: deciding what a system should be allowed to do on its own in the first place, and how people can understand, change and reverse what it did.

Autonomy isn’t one switch

Many AI products treat autonomy as a single setting: the assistant is either in “suggest” mode or “agent” mode. Human factors researchers worked out a more useful model more than twenty years ago. Raja Parasuraman, Thomas Sheridan and Christopher Wickens split any automated task into four stages: gathering information, analysing it, choosing what to do, and doing it. Each stage can be automated to a different degree, on a scale that runs from the computer offering no help at all to the computer acting entirely on its own and not telling anyone.

What I like about this model is that it makes autonomy a design decision for each action, not a personality for the whole product. An assistant can be very autonomous at gathering and sorting your email while being barely autonomous at sending it. That’s not inconsistency. It reflects the fact that sorting is easy to undo and sending isn’t.

When I think about where a specific action belongs on that scale, I ask three questions:

• Can it be undone, and by whom? Moving a label can be reversed in a click. A sent message, a payment or a deleted file often can’t.
• Who else does it touch? Rescheduling your own focus time affects you. Rescheduling a meeting notifies six other people, who will act on it.
• What does a mistake cost? Not on average, but in the worst plausible case.

The answers usually differ between actions inside the same product, which is why one global “autonomy” toggle almost always ends up wrong for something.

When to recommend, and when to act

In 1999 Eric Horvitz described what he called mixed-initiative interfaces, where the person and the system both take turns leading. His key idea was that a system should compare the expected value of three options: acting, asking first, or doing nothing. That depends on how sure the system is about what the person wants. When it’s unsure, asking is worth the interruption. When it’s confident and the action is cheap, asking is just friction.

I’d add a second variable that matters more now that systems can act in the world: the cost of being wrong. That cost depends on reversibility. So my rule of thumb is simple. The harder an action is to undo, the more certain the system needs to be before it acts, and some actions should never cross that line at all. Sending a message in your name is one of them for me. A system can draft it, but you send it.

In practice this means an agent should show its plan before it runs it, and mark which steps can be reversed. If a plan mixes reversible and irreversible steps, the interface can let the person approve the safe ones in a single action and slow down only for the rest. That’s more honest than one big “Go” button, and it puts the person’s attention where it’s actually needed.

Saying how sure it is, without a percentage

Microsoft researchers led by Saleema Amershi pulled together 18 guidelines for human-AI interaction and tested them against 20 real AI products with 49 design practitioners. Several of them are about exactly this moment: make clear how well the system can do what it does, support efficient correction, and scope the service down when in doubt.

The obvious way to communicate uncertainty is a number: “87% confident”. I don’t think that works well for actions. Most people don’t know what to do with 87%. Is that good? Should they check? Matthew Kay and colleagues found that how uncertainty is shown makes a real difference. In their study of bus arrival predictions, showing the spread of likely arrival times as a set of dots helped people judge their chances of catching the bus more accurately than other displays did. The lesson I take from it is to show uncertainty in terms of the decision the person is actually making.

For an agent, the decision is usually “should I let this happen?”. So instead of a score, I’d rather show what the system checked, what matched and what didn’t. Two things line up, one thing is off by nine days: that’s something a person can verify in seconds. It also avoids a trap I mentioned in my last post on this topic, where explanations make people more likely to accept a recommendation whether it’s right or not. A list of checks the person can test works differently from a paragraph of reasons.

Designing for undo

If I had to pick one design principle for systems that act, it would be reversibility. Undo turns a decision from something people have to get right in advance into something they can correct afterwards, and it’s much easier to oversee a system you can correct. Gmail’s “Undo Send” is the familiar example: a short delay that turns an irreversible action into a reversible one, at almost no cost.

There are a few patterns I reach for:

• Preview before acting. Show exactly what will change, who will be notified and what will be sent, before anything happens.
• Stage instead of send. Prepare drafts, holds and pending changes, and let the person release them.
• A short undo window. For actions that are safe to delay, a few seconds of “undo” covers most mistakes.
• A readable log. “What did it do while I wasn’t looking?” should have a plain answer, listed by action, with an undo next to each one that still allows it.
• A repair path when undo is impossible. You can’t unsend an email, but the system can help you send a correction quickly.

Consent fits into this too. Permission to act shouldn’t be a checkbox people tick during onboarding and never see again. It should be scoped to specific actions, limited in time when that makes sense, easy to take back, and asked for at the moment it matters, the way phones ask for location access when an app first needs it rather than at install.

When the decision is what a group thinks

The hardest version of this problem I’ve worked on is in Between, a decision tool for teams. Everyone answers privately first, then the team sees where it agrees and where it splits, and the decision is recorded with the disagreement still attached. Between doesn’t use AI today. In my research I sorted every answer by hand, and the product shows counts next to the reasons behind them.

But the obvious next feature is a summary. Forty people write four answers each, and a model turns them into one paragraph about what the team thinks. It would save a lot of reading. It would also quietly hand the system one of the most important decisions in the process: what the group believes.

I think that’s risky for three reasons. First, groups already have a well-documented tendency to discuss what most members share and to drop what only one or two people know. Garold Stasser and William Titus showed this in 1985, and it’s a big part of why Between exists. A summary that favours the most common points does the same thing, faster, and the minority view is by definition the least common. Second, summaries aren’t always faithful. Joshua Maynez and colleagues found that abstractive summarisation models frequently added information that wasn’t in the source at all. Models have improved since, but a summary of opinions is exactly where a small distortion matters. Third, wording shapes belief. Maurice Jakesch and colleagues found that people who wrote with a language model that leaned toward one view shifted toward that view, both in what they wrote and in a later survey of their attitudes. A summary isn’t a neutral container.

So if I added summaries to Between, I would design them like this:

• The summary is a draft. Everyone who answered can read it before it’s recorded, and the decision owner, not the model, records the outcome.
• Every line is traceable. Each sentence shows how many answers it stands for and links to them.
• Disagreement can’t be summarised away. “Still disagree” is a required part of the summary, even when only two people hold that view.
• People can say “that’s not what I meant”. Flagging a line sends it back for review instead of disappearing into a feedback form.
• The privacy rules don’t change. Between shows nothing until five people have answered, and the summary follows the same rule.

None of that makes the summary less useful. It makes it something the team can check, rather than something the team has to trust.

A system should make the group easier to hear, not decide what the group said.

What meaningful oversight looks like

“Human oversight” can mean almost anything, so I find it useful to be specific. For me, a person has meaningful oversight of a system that acts on their behalf when they can do four things: see what it’s about to do, understand why in terms they can check, change it before it happens, and reverse it, or at least repair it, afterwards. If any of those is missing, the person is supervising in name only.

Most of that is design work, not model work. Better models will make fewer mistakes, but they won’t decide where the line between “suggest” and “act” sits, how uncertainty is shown, or whether there’s an undo button. Those are product decisions, and I think they’ll matter more as systems get more capable, not less. The more a system can do, the more it matters that people can still tell it no.

References

7

1. Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R. & Horvitz, E. (2019). Guidelines for human-AI interaction. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13. doi.org/10.1145/3290605.3300233
2. Horvitz, E. (1999). Principles of mixed-initiative user interfaces. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 159–166. doi.org/10.1145/302979.303030
3. Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L. & Naaman, M. (2023). Co-writing with opinionated language models affects users’ views. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–15. doi.org/10.1145/3544548.3581196
4. Kay, M., Kola, T., Hullman, J. R. & Munson, S. A. (2016). When (ish) is my bus? User-centered visualizations of uncertainty in everyday, mobile predictive systems. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 5092–5103. doi.org/10.1145/2858036.2858558
5. Maynez, J., Narayan, S., Bohnet, B. & McDonald, R. (2020). On faithfulness and factuality in abstractive summarization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 1906–1919. doi.org/10.18653/v1/2020.acl-main.173
6. Parasuraman, R., Sheridan, T. B. & Wickens, C. D. (2000). A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics, Part A, 30(3), 286–297. doi.org/10.1109/3468.844354
7. Stasser, G. & Titus, W. (1985). Pooling of unshared information in group decision making: Biased information sampling during discussion. Journal of Personality and Social Psychology, 48(6), 1467–1478. doi.org/10.1037/0022-3514.48.6.1467