When Should Humans Step In? Optimal Human Dispatching in AI-Assisted Decisions

TL;DR

This paper introduces a decision-theoretic framework for optimizing when humans should intervene in AI-assisted decision making, improving efficiency and accuracy in tasks like peer review.

Contribution

It develops a general framework for human-AI collaboration that models human judgments as costly information and provides estimation procedures for optimal intervention policies.

Findings

Outperforms LLM-only predictions in peer review tasks.

Achieves similar accuracy to full human review with only 20-30% human effort.

Simple linear models can reduce computational costs without sacrificing performance.

Abstract

AI systems increasingly assist human decision making by producing preliminary assessments of complex inputs. However, such AI-generated assessments can often be noisy or systematically biased, raising a central question: how should costly human effort be allocated to correct AI outputs where it matters the most for the final decision? We propose a general decision-theoretic framework for human-AI collaboration in which AI assessments are treated as factor-level signals and human judgments as costly information that can be selectively acquired. We consider cases where the optimal selection problem reduces to maximizing a reward associated with each candidate subset of factors, and turn policy design into reward estimation. We develop estimation procedures under both nonparametric and linear models, covering contextual and non-contextual selection rules. In the linear setting, the optimal rule admits a closed-form expression with a clear interpretation in terms of factor importance and residual variance. We apply our framework to AI-assisted peer review. Our approach substantially outperforms LLM-only predictions and achieves performance comparable to full human review while using only 20-30% of the human information. Across different selection rules, we find that simpler rules derived under linear models can significantly reduce computational cost without harming final prediction performance. Our results highlight both the value of human intervention and the efficiency of principled dispatching.