The right to be unoptimized

Ask people how they found the things they love most and the answers are often accidents. A book picked up because the one they wanted was out. A friendship that started because a train was cancelled. A career that began with a class taken to fill a gap in a timetable. None of these would have been recommended, because none of them matched anything the person had done before.

Products are getting very good at recommending what matches what we’ve done before. With AI, they can now plan the week, pick dinner, tune playlists and suggest the next purchase, each one a little better than we would have managed. Every single suggestion is an improvement. I’m less sure about the total.

What optimization quietly removes

An optimizer needs a goal. Fastest route, highest rating, most likely to click. Whatever isn’t part of the goal slowly disappears, not because anyone decided to remove it, but because it never wins.

On a map app, that’s the slower street with the good bakery. In a music app, it’s the genre you’ve never tried. In a calendar, it’s the empty hour. These things have value, but their value is hard to predict, so a system built on predictions keeps choosing something else.

People who build recommender systems know this as the trade-off between exploitation (giving you more of what works) and exploration (trying something uncertain). Most commercial systems lean hard toward exploitation, because it shows up in this quarter’s numbers. Exploration pays off later, and sometimes for the user rather than the business.

There’s a further problem, which is that the things we lose are hard to even name. Denis Kotkov and colleagues, reviewing research on serendipity in recommender systems, found that the field doesn’t agree on what serendipity is. Most definitions combine some mix of relevance, novelty and unexpectedness, and each of those is measured differently from paper to paper. That matters more than it sounds. A goal you can’t define is a goal you can’t put on a dashboard, and a goal that isn’t on the dashboard loses every argument to one that is.

The loop nobody designed

The narrowing isn’t only a matter of priorities. Some of it is built into how these systems learn.

A recommender learns from what you click. But you can only click on what it showed you. So the data it learns from tomorrow is shaped by the choices it made today. Allison Chaney, Brandon Stewart and Barbara Engelhardt call this algorithmic confounding. In simulations, they showed that training on data produced this way makes users’ behaviour more alike without making them better off. The system grows more confident about a smaller world, and reads its own influence as evidence of what people want.

I find this the most useful way to think about the problem, because it removes the villain. Nobody has to decide to narrow anyone’s life. Narrowing is simply what a feedback loop does when its main teacher is its own output.

The best real-world evidence I know of comes from Spotify. Ashton Anderson and colleagues measured how musically diverse millions of listeners were, and found two things that pull against each other. Listeners with more diverse habits were much more likely to stay and to become paying subscribers. But listening driven by algorithmic recommendations was associated with less diversity, and people who became more diverse over time tended to do it by moving away from recommendations and finding music on their own. Most of this is correlational, so it doesn’t show the recommendations caused the narrowing. It does show the tension clearly: the habit linked to people staying is the one the algorithm seems least able to grow.

That changes the argument for exploration. It stops looking like a favour to users at the expense of the business, and starts looking like an investment the business is under-making because its metrics can’t see the return.

It isn’t as simple as “algorithms trap us”

I want to be careful here, because the popular version of this story, the filter bubble, is tidier than the evidence.

Tien Nguyen and colleagues followed users of MovieLens, a film recommendation site, over time. The recommendations people saw did narrow slightly as they kept rating films. But here is the surprising part: people who actually watched what was recommended to them narrowed less than people who didn’t, and rated what they watched more highly. Some narrowing seems to be what happens anyway as a person works out what they like. Left alone, we can be quite good at building our own small rooms.

So I don’t think the honest position is “recommendations are bad”. It’s closer to this: recommendations are a powerful way of steering, and the question is what they steer toward and who gets to hold the wheel. A system tuned for the next click will steer one way. A system tuned for how someone’s taste develops over a year could steer quite differently, and might even widen it.

Inefficiency isn’t always waste

A lot of human value lives in the inefficient parts. Browsing is inefficient compared with searching, and it’s how we find things we didn’t know to look for. Cooking without a recipe is slower and teaches you how flavours work. Wandering is a terrible way to get somewhere and a great way to learn a city.

Psychology has a name for the opposite approach. Barry Schwartz and colleagues distinguished maximizers, people who try to find the best possible option, from satisficers, who look until they find something good enough and stop. Across several samples, maximizers reported less happiness and life satisfaction and more regret. In a later study, Sheena Iyengar, Rachael Wells and Schwartz followed graduating students through their job search. The maximizers ended up with starting salaries about 20% higher, and were less satisfied with the jobs they took and felt worse throughout the search. The authors linked this to maximizers’ fixation on the options they didn’t pick.

This is the part that worries me about optimization as a product posture. When an interface constantly ranks, scores and compares, it nudges everyone into the maximizer’s position: there is always a better option, and you can always see it. You may get the 20%. You may also get the regret.

One honest caveat. The popular idea that more choice always makes people miserable hasn’t held up well. When Benjamin Scheibehenne, Rainer Greifeneder and Peter Todd pooled 50 experiments on “choice overload”, the average effect was close to zero, with a lot of variation between studies. So the problem isn’t the number of options. It’s the stance of treating every choice as something to be optimized.

There’s also something about autonomy. A choice that’s been optimized for you, however good, isn’t quite a choice you made. Over time, a life where most choices arrive pre-made can start to feel less like your own.

Quentin André and colleagues, in a review drawing on marketing, economics, philosophy and psychology, argue that automation and personalization create exactly this tension: they make choosing easier and more efficient, and they can also weaken people’s sense that their choices are their own, which matters for wellbeing. It’s a framing paper that sets out open questions rather than an experiment, but I think it names the trade-off well. Michael Ekstrand and Martijn Willemsen make a related point from inside the field: watching what people click isn’t the same as knowing what they want. They argue that recommenders get better when they listen to what people say about their goals and give them real control, rather than relying on behaviour alone.

Not every moment needs to be improved. Some just need to be left alone.

What designing for this could look like

I don’t think the answer is to throw away recommendations. They’re useful, and most people want them most of the time. But designers can build in room to be unoptimized:

• A visible dial. Let people choose between “more like this” and “surprise me”, and keep the setting where they can see it, not buried in preferences. A dial also turns exploration from something the system does to you into something you ask for, which is the autonomy point in one control.
• Protected blank space. A scheduling assistant that leaves some time unfilled on purpose, and treats it as protected, not as a gap to solve.
• Honest randomness. When something is a wildcard, say so. “This is outside your usual taste” makes the surprise feel like a gift, not a mistake. It also stops a miss from being read as a failure, by the person or by the model.
• Ask, don’t just watch. Clicks tell you what someone did with what you showed them. Now and then, ask what they’re trying to get out of the product. The answer is information the feedback loop can never produce by itself.
• Good-enough modes. Not every decision needs a ranked list. Sometimes the kinder design is one solid option and a way to move on, instead of ten scored ones and a nagging sense of the other nine.
• The off switch. Let people turn personalization off for a while without losing their history.
• Success measured over months, not sessions. If a product only counts this week’s clicks, it will always pick the safe option. The Spotify findings suggest the long view isn’t only kinder; it may be where the value is.

The tension I can’t fully resolve

There’s a trap in all of this, and I don’t want to pretend I’ve escaped it. If a team decides serendipity is good and starts measuring it, they’ve built a new optimizer. An optimizer for surprise is still an optimizer. Before long, the “unexpected” picks are as predictable as everything else, and the empty hour on the calendar is a feature with a target.

There’s a second tension too. Deciding that people need more randomness than they choose is its own kind of paternalism. Someone who wants the same comfort show every night isn’t doing life wrong.

The way through, I think, is to stop treating this as a question of the right amount of exploration and treat it as a question of control. The aim isn’t to make people less optimized. It’s to make sure the optimizing is something they can see, steer and switch off, so that the unplanned parts of life stay theirs to plan or not.

A different kind of helpful

We usually define a helpful product as one that saves time and effort. That’s a fine definition for tasks. But people aren’t only a list of tasks. Sometimes the helpful thing is to step back, show something unexpected, or leave space and trust the person to fill it.

As AI takes on more of the planning in people’s lives, I think “leave me some room” will become a real design requirement, the same way privacy did. The products that respect it may lose a few points on a dashboard. I think they’ll be the ones people still trust in ten years.

References

9

1. Anderson, A., Maystre, L., Anderson, I., Mehrotra, R. & Lalmas, M. (2020). Algorithmic effects on the diversity of consumption on Spotify. In Proceedings of The Web Conference 2020 (pp. 2155–2165). ACM. doi.org/10.1145/3366423.3380281
2. André, Q., Carmon, Z., Wertenbroch, K., Crum, A., Frank, D., Goldstein, W., Huber, J., van Boven, L., Weber, B. & Yang, H. (2018). Consumer choice and autonomy in the age of artificial intelligence and big data. Customer Needs and Solutions, 5(1–2), 28–37. doi.org/10.1007/s40547-017-0085-8
3. Chaney, A. J. B., Stewart, B. M. & Engelhardt, B. E. (2018). How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems (pp. 224–232). ACM. doi.org/10.1145/3240323.3240370
4. Ekstrand, M. D. & Willemsen, M. C. (2016). Behaviorism is not enough: Better recommendations through listening to users. In Proceedings of the 10th ACM Conference on Recommender Systems (pp. 221–224). ACM. doi.org/10.1145/2959100.2959179
5. Iyengar, S. S., Wells, R. E. & Schwartz, B. (2006). Doing better but feeling worse: Looking for the “best” job undermines satisfaction. Psychological Science, 17(2), 143–150. doi.org/10.1111/j.1467-9280.2006.01677.x
6. Kotkov, D., Wang, S. & Veijalainen, J. (2016). A survey of serendipity in recommender systems. Knowledge-Based Systems, 111, 180–192. doi.org/10.1016/j.knosys.2016.08.014
7. Nguyen, T. T., Hui, P.-M., Harper, F. M., Terveen, L. & Konstan, J. A. (2014). Exploring the filter bubble: The effect of using recommender systems on content diversity. In Proceedings of the 23rd International Conference on World Wide Web (pp. 677–686). ACM. doi.org/10.1145/2566486.2568012
8. Scheibehenne, B., Greifeneder, R. & Todd, P. M. (2010). Can there ever be too many options? A meta-analytic review of choice overload. Journal of Consumer Research, 37(3), 409–425. doi.org/10.1086/651235
9. Schwartz, B., Ward, A., Monterosso, J., Lyubomirsky, S., White, K. & Lehman, D. R. (2002). Maximizing versus satisficing: Happiness is a matter of choice. Journal of Personality and Social Psychology, 83(5), 1178–1197. doi.org/10.1037/0022-3514.83.5.1178