Budgeted Multiple-Expert Deferral

TL;DR

This paper introduces a budgeted deferral framework for machine learning that selectively queries experts during training to reduce costs while maintaining accuracy, supported by theoretical guarantees and empirical evidence.

Contribution

It presents novel algorithms for budgeted expert deferral that minimize training costs and provides theoretical analysis and empirical validation of their effectiveness.

Findings

Significant reduction in training expert query costs.

Maintained or improved predictive accuracy.

Theoretical guarantees on generalization and label complexity.

Abstract

Learning to defer uncertain predictions to costly experts offers a powerful strategy for improving the accuracy and efficiency of machine learning systems. However, standard training procedures for deferral algorithms typically require querying all experts for every training instance, an approach that becomes prohibitively expensive when expert queries incur significant computational or resource costs. This undermines the core goal of deferral: to limit unnecessary expert usage. To overcome this challenge, we introduce the budgeted deferral framework, which aims to train effective deferral algorithms while minimizing expert query costs during training. We propose new algorithms for both two-stage and single-stage multiple-expert deferral settings that selectively query only a subset of experts per training example. While inspired by active learning, our setting is fundamentally different: labels are already known, and the core challenge is to decide which experts to query in order to balance cost and predictive performance. We establish theoretical guarantees for both of our algorithms, including generalization bounds and label complexity analyses. Empirical results across several domains show that our algorithms substantially reduce training costs without sacrificing prediction accuracy, demonstrating the practical value of our budget-aware deferral algorithms.