When More Experts Hurt: Underfitting in Multi-Expert Learning to Defer

TL;DR

This paper reveals that multi-expert learning to defer faces inherent underfitting issues due to expert identifiability problems, and proposes PiCCE, a method to effectively select reliable experts, improving prediction accuracy.

Contribution

The paper identifies a fundamental challenge in multi-expert learning to defer and introduces PiCCE, a surrogate method that addresses expert underfitting by adaptively selecting reliable experts.

Findings

PiCCE reduces multi-expert L2D to a single-expert-like problem.

PiCCE is statistically consistent and recovers class probabilities.

Experiments show PiCCE outperforms existing methods across diverse scenarios.

Abstract

Learning to Defer (L2D) enables a classifier to abstain from predictions and defer to an expert, and has recently been extended to multi-expert settings. In this work, we show that multi-expert L2D is fundamentally more challenging than the single-expert case. With multiple experts, the classifier's underfitting becomes inherent, which seriously degrades prediction performance, whereas in the single-expert setting it arises only under specific conditions. We theoretically reveal that this stems from an intrinsic expert identifiability issue: learning which expert to trust from a diverse pool, a problem absent in the single-expert case and renders existing underfitting remedies failed. To tackle this issue, we propose PiCCE (Pick the Confident and Correct Expert), a surrogate-based method that adaptively identifies a reliable expert based on empirical evidence. PiCCE effectively reduces multi-expert L2D to a single-expert-like learning problem, thereby resolving multi expert underfitting. We further prove its statistical consistency and ability to recover class probabilities and expert accuracies. Extensive experiments across diverse settings, including real-world expert scenarios, validate our theoretical results and demonstrate improved performance.