Robust Multi-Task Learning with Excess Risks

TL;DR

This paper introduces ExcessMTL, a novel multi-task learning method that balances tasks based on their excess risks, improving robustness against label noise and outperforming existing methods.

Contribution

We propose ExcessMTL, an excess risk-based task weighting scheme with theoretical guarantees and practical effectiveness in noisy label scenarios.

Findings

ExcessMTL outperforms existing methods on various benchmarks.

The algorithm achieves convergence guarantees and Pareto stationarity.

Demonstrates robustness to label noise in multi-task learning.

Abstract

Multi-task learning (MTL) considers learning a joint model for multiple tasks by optimizing a convex combination of all task losses. To solve the optimization problem, existing methods use an adaptive weight updating scheme, where task weights are dynamically adjusted based on their respective losses to prioritize difficult tasks. However, these algorithms face a great challenge whenever label noise is present, in which case excessive weights tend to be assigned to noisy tasks that have relatively large Bayes optimal errors, thereby overshadowing other tasks and causing performance to drop across the board. To overcome this limitation, we propose Multi-Task Learning with Excess Risks (ExcessMTL), an excess risk-based task balancing method that updates the task weights by their distances to convergence instead. Intuitively, ExcessMTL assigns higher weights to worse-trained tasks that are further from convergence. To estimate the excess risks, we develop an efficient and accurate method with Taylor approximation. Theoretically, we show that our proposed algorithm achieves convergence guarantees and Pareto stationarity. Empirically, we evaluate our algorithm on various MTL benchmarks and demonstrate its superior performance over existing methods in the presence of label noise. Our code is available at https://github.com/yifei-he/ExcessMTL.