Human-AI Collaborative Uncertainty Quantification

TL;DR

This paper introduces a formal framework for human-AI collaborative uncertainty quantification, improving decision-making reliability by combining human expertise with AI predictions through novel calibration algorithms and theoretical insights.

Contribution

It formalizes the principles of human-AI collaboration in uncertainty quantification, extending conformal prediction, and develops practical algorithms with guarantees for adaptive, distribution-shifting environments.

Findings

Collaborative prediction sets outperform individual agents in coverage and size.

The optimal prediction structure follows a two-threshold rule over a score function.

Algorithms adapt to distribution shifts, including human-AI interaction dynamics.

Abstract

AI predictive systems are increasingly embedded in decision making pipelines, shaping high stakes choices once made solely by humans. Yet robust decisions under uncertainty still rely on capabilities that current AI lacks: domain knowledge not captured by data, long horizon context, and reasoning grounded in the physical world. This gap has motivated growing efforts to design collaborative frameworks that combine the complementary strengths of humans and AI. This work advances this vision by identifying the fundamental principles of Human AI collaboration within uncertainty quantification, a key component of reliable decision making. We introduce Human AI Collaborative Uncertainty Quantification, a framework that formalizes how an AI model can refine a human expert's proposed prediction set with two goals: avoiding counterfactual harm, ensuring the AI does not degrade correct human judgments, and complementarity, enabling recovery of correct outcomes the human missed. At the population level, we show that the optimal collaborative prediction set follows an intuitive two threshold structure over a single score function, extending a classical result in conformal prediction. Building on this insight, we develop practical offline and online calibration algorithms with provable distribution free finite sample guarantees. The online method adapts to distribution shifts, including human behavior evolving through interaction with AI, a phenomenon we call Human to AI Adaptation. Experiments across image classification, regression, and text based medical decision making show that collaborative prediction sets consistently outperform either agent alone, achieving higher coverage and smaller set sizes across various conditions.