Coverage-Constrained Human-AI Cooperation with Multiple Experts

TL;DR

This paper introduces CL2DC, a novel method for human-AI cooperation that optimally balances AI and expert decisions under coverage constraints, especially in noisy-label scenarios, improving decision accuracy in high-stakes tasks.

Contribution

The paper proposes CL2DC, a coverage-constrained learning approach that effectively integrates multiple experts and AI predictions, addressing a key gap in human-AI cooperation methods.

Findings

CL2DC outperforms existing methods on synthetic datasets.

It achieves superior accuracy on real-world datasets.

The approach effectively manages noisy labels without clean references.

Abstract

Human-AI cooperative classification (HAI-CC) approaches aim to develop hybrid intelligent systems that enhance decision-making in various high-stakes real-world scenarios by leveraging both human expertise and AI capabilities. Current HAI-CC methods primarily focus on learning-to-defer (L2D), where decisions are deferred to human experts, and learning-to-complement (L2C), where AI and human experts make predictions cooperatively. However, a notable research gap remains in effectively exploring both L2D and L2C under diverse expert knowledge to improve decision-making, particularly when constrained by the cooperation cost required to achieve a target probability for AI-only selection (i.e., coverage). In this paper, we address this research gap by proposing the Coverage-constrained Learning to Defer and Complement with Specific Experts (CL2DC) method. CL2DC makes final decisions through either AI prediction alone or by deferring to or complementing a specific expert, depending on the input data. Furthermore, we propose a coverage-constrained optimisation to control the cooperation cost, ensuring it approximates a target probability for AI-only selection. This approach enables an effective assessment of system performance within a specified budget. Also, CL2DC is designed to address scenarios where training sets contain multiple noisy-label annotations without any clean-label references. Comprehensive evaluations on both synthetic and real-world datasets demonstrate that CL2DC achieves superior performance compared to state-of-the-art HAI-CC methods.