Density-Ratio Losses for Post-Hoc Learning to Defer

TL;DR

This paper introduces a novel density-ratio loss framework for post-hoc Learning to Defer, enabling adjustable deferral decisions based on ideal distribution reweightings, with theoretical insights and competitive experimental results.

Contribution

It develops DR CPE losses for post-hoc L2D, connecting deferral rules to density ratios and ideal distributions, and demonstrates robustness and theoretical links to existing methods.

Findings

Our approach is competitive with common baselines.

It offers more robust performance across datasets.

Connects deferral rules to density-ratio estimation and ideal distributions.

Abstract

We study post-hoc Learning to Defer (L2D) through the lens of ideal distributions: divergence-regularized reweightings of the data distribution under which a model attains low loss. We define deferral via the density-ratio between a model's and an expert's ideals. Using the reduction from density-ratio estimation to class-probability estimation, we derive the DR CPE losses for post-hoc L2D scorers. Deferral decisions are then made by thresholding the scorer, allowing deferral rates to be adjusted without retraining. For KL-based ideal distributions, our deferral rules recovers Chow's rule under the original distribution and a connection to an expert-tilted Bayes posterior -- which incorporates the expert's performance -- depending on if the ideal distributions are joint or marginal distributions. Experimentally, our approach is competitive compared to common baselines and more robust across dataset settings. More broadly, our results cast post-hoc L2D as density-ratio learning between ideal distributions, bridging Chow-style rules, expert comparison, and elucidating connections to related learning settings including anomaly detection.