Human-AI Collaboration with Misaligned Preferences

TL;DR

This paper investigates human-AI collaboration where preferences are noisy and misaligned, revealing that humans can benefit more from algorithms with different mistakes than from perfectly aligned ones, with implications for design and policy.

Contribution

The paper provides a theoretical analysis of misaligned human-AI collaboration, showing that misaligned algorithms can sometimes enhance human utility and exploring properties that maximize welfare.

Findings

Humans benefit from misaligned algorithms more than from aligned ones.

Misaligned algorithms can improve human utility under certain conditions.

Design principles for algorithms that maximize human welfare are discussed.

Abstract

In many real-life settings, algorithms play the role of assistants, while humans ultimately make the final decision. Often, algorithms specifically act as curators, narrowing down a wide range of options into a smaller subset that the human picks between: consider content recommendation or chatbot responses to questions with multiple valid answers. Crucially, humans may not know their own preferences perfectly either, but instead may only have access to a noisy sampling over preferences. Algorithms can assist humans by curating a smaller subset of items, but must also face the challenge of misalignment: humans may have different preferences from each other (and from the algorithm), and the algorithm may not know the exact preferences of the human they are facing at any point in time. In this paper, we model and theoretically study such a setting. Specifically, we show instances where humans benefit by collaborating with a misaligned algorithm. Surprisingly, we show that humans gain more utility from a misaligned algorithm (which makes different mistakes) than from an aligned algorithm. Next, we build on this result by studying what properties of algorithms maximize human welfare when the goals could be either utilitarian welfare or ensuring all humans benefit. We conclude by discussing implications for designers of algorithmic tools and policymakers.