When Are Two Lists Better than One?: Benefits and Harms in Joint Decision-making

TL;DR

This paper investigates when joint decision-making between humans and algorithms outperforms individual choices, revealing that collaboration can be beneficial under various noise models, but can also be detrimental depending on the interaction dynamics.

Contribution

It provides a theoretical and experimental analysis of optimal subset sizes in human-algorithm collaboration, highlighting conditions where collaboration improves or worsens performance.

Findings

Optimal subset size often between 2 and n-1 under noise models.

Collaboration benefits depend on the nature of human-algorithm interaction.

Performance can be worse when the human is anchored on the algorithm's ordering.

Abstract

Historically, much of machine learning research has focused on the performance of the algorithm alone, but recently more attention has been focused on optimizing joint human-algorithm performance. Here, we analyze a specific type of human-algorithm collaboration where the algorithm has access to a set of $n$ items, and presents a subset of size $k$ to the human, who selects a final item from among those $k$. This scenario could model content recommendation, route planning, or any type of labeling task. Because both the human and algorithm have imperfect, noisy information about the true ordering of items, the key question is: which value of $k$ maximizes the probability that the best item will be ultimately selected? For $k=1$, performance is optimized by the algorithm acting alone, and for $k=n$ it is optimized by the human acting alone. Surprisingly, we show that for multiple of noise models, it is optimal to set $k \in [2, n-1]$ - that is, there are strict benefits to collaborating, even when the human and algorithm have equal accuracy separately. We demonstrate this theoretically for the Mallows model and experimentally for the Random Utilities models of noisy permutations. However, we show this pattern is reversed when the human is anchored on the algorithm's presented ordering - the joint system always has strictly worse performance. We extend these results to the case where the human and algorithm differ in their accuracy levels, showing that there always exist regimes where a more accurate agent would strictly benefit from collaborating with a less accurate one, but these regimes are asymmetric between the human and the algorithm's accuracy.