Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity

TL;DR

This paper introduces label-distributionally robust (LDR) losses for multi-class classification, providing a unified framework that explains classical losses, achieves adaptivity to label uncertainty, and demonstrates strong empirical performance under noisy labels.

Contribution

It establishes top-$k$ consistency results for LDR losses, proposes an adaptive LDR loss with instance-level noise adaptation, and shows competitive empirical results on benchmark datasets.

Findings

LDR losses unify classical loss functions and their variants.

The adaptive LDR loss improves robustness to label noise.

Empirical results show stable, competitive performance under noisy conditions.

Abstract

We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification that are formulated from distributionally robust optimization (DRO) perspective, where the uncertainty in the given label information are modeled and captured by taking the worse case of distributional weights. The benefits of this perspective are several fold: (i) it provides a unified framework to explain the classical cross-entropy (CE) loss and SVM loss and their variants, (ii) it includes a special family corresponding to the temperature-scaled CE loss, which is widely adopted but poorly understood; (iii) it allows us to achieve adaptivity to the uncertainty degree of label information at an instance level. Our contributions include: (1) we study both consistency and robustness by establishing top-$k$ ($\forall k\geq 1$) consistency of LDR losses for multi-class classification, and a negative result that a top-$1$ consistent and symmetric robust loss cannot achieve top-$k$ consistency simultaneously for all $k\geq 2$; (2) we propose a new adaptive LDR loss that automatically adapts the individualized temperature parameter to the noise degree of class label of each instance; (3) we demonstrate stable and competitive performance for the proposed adaptive LDR loss on 7 benchmark datasets under 6 noisy label and 1 clean settings against 13 loss functions, and on one real-world noisy dataset. The code is open-sourced at \url{https://github.com/Optimization-AI/ICML2023_LDR}.