A No Free Lunch Theorem for Human-AI Collaboration

TL;DR

This paper proves that in binary classification, any collaborative strategy that tries to outperform individual agents without always deferring to one will sometimes perform worse, highlighting fundamental limits in human-AI collaboration.

Contribution

It establishes a No Free Lunch theorem for human-AI collaboration, showing inherent limitations and conditions for effective combined decision-making.

Findings

Deterministic collaboration strategies can underperform compared to the least accurate agent.

Always deferring to the most accurate agent can be optimal in some cases.

Guides the design of human-AI collaboration methods based on theoretical limits.

Abstract

The gold standard in human-AI collaboration is complementarity -- when combined performance exceeds both the human and algorithm alone. We investigate this challenge in binary classification settings where the goal is to maximize 0-1 accuracy. Given two or more agents who can make calibrated probabilistic predictions, we show a "No Free Lunch"-style result. Any deterministic collaboration strategy (a function mapping calibrated probabilities into binary classifications) that does not essentially always defer to the same agent will sometimes perform worse than the least accurate agent. In other words, complementarity cannot be achieved "for free." The result does suggest one model of collaboration with guarantees, where one agent identifies "obvious" errors of the other agent. We also use the result to understand the necessary conditions enabling the success of other collaboration techniques, providing guidance to human-AI collaboration.