Optimized Deferral for Imbalanced Settings

TL;DR

This paper introduces MILD, a new deferral algorithm that addresses expert imbalance in learning to defer, improving performance in image classification and LLM routing tasks.

Contribution

It develops margin-based loss functions and algorithms tailored for expert imbalance, advancing the learning to defer methodology.

Findings

MILD outperforms existing baselines in image classification tasks.

MILD improves LLM routing accuracy in real-world scenarios.

The proposed loss functions provide theoretical guarantees for imbalanced settings.

Abstract

Learning algorithms can be significantly improved by routing complex or uncertain inputs to specialized experts, balancing accuracy with computational cost. This approach, known as learning to defer, is essential in domains like natural language generation, medical diagnosis, and computer vision, where an effective deferral can reduce errors at low extra resource consumption. However, the two-stage learning to defer setting, which leverages existing predictors such as a collection of LLMs or other classifiers, often faces challenges due to an expert imbalance problem. This imbalance can lead to suboptimal performance, with deferral algorithms favoring the majority expert. We present a comprehensive study of two-stage learning to defer in expert imbalance settings. We cast the deferral loss optimization as a novel cost-sensitive learning problem over the input-expert domain. We derive new margin-based loss functions and guarantees tailored to this setting, and develop novel algorithms for cost-sensitive learning. Leveraging these results, we design principled deferral algorithms, MILD (Margin-based Imbalanced Learning to Defer), specifically suited for expert imbalance settings. Extensive experiments demonstrate the effectiveness of our approach, showing clear improvements over existing baselines on both image classification and real-world Large Language Model (LLM) routing tasks.