A Two-Stage Learning-to-Defer Approach for Multi-Task Learning

TL;DR

This paper introduces a novel two-stage Learning-to-Defer framework for multi-task learning that jointly handles classification and regression, providing theoretical guarantees and demonstrating effectiveness on real-world tasks.

Contribution

The paper proposes a unified two-stage L2D framework for multi-task learning with theoretical consistency guarantees and practical validation on complex tasks.

Findings

Effective joint classification and regression with L2D

Theoretical guarantees for convergence to optimal rejector

Outperforms existing L2D methods in multi-task scenarios

Abstract

The Two-Stage Learning-to-Defer (L2D) framework has been extensively studied for classification and, more recently, regression tasks. However, many real-world applications require solving both tasks jointly in a multi-task setting. We introduce a novel Two-Stage L2D framework for multi-task learning that integrates classification and regression through a unified deferral mechanism. Our method leverages a two-stage surrogate loss family, which we prove to be both Bayes-consistent and $(\mathcal{G}, \mathcal{R})$-consistent, ensuring convergence to the Bayes-optimal rejector. We derive explicit consistency bounds tied to the cross-entropy surrogate and the $L_1$-norm of agent-specific costs, and extend minimizability gap analysis to the multi-expert two-stage regime. We also make explicit how shared representation learning -- commonly used in multi-task models -- affects these consistency guarantees. Experiments on object detection and electronic health record analysis demonstrate the effectiveness of our approach and highlight the limitations of existing L2D methods in multi-task scenarios.