Tackling Instance-Dependent Label Noise with Dynamic Distribution Calibration

TL;DR

This paper introduces a dynamic distribution calibration approach to address instance-dependent label noise by modeling feature distributions as Gaussians and calibrating shifts caused by label corruption, improving model robustness.

Contribution

It proposes two novel methods based on Gaussian mean and covariance for distribution calibration, with theoretical guarantees and experimental validation against label noise.

Findings

The mean-based method robustly estimates class means under label noise.

The covariance-based method effectively improves model robustness.

Both methods outperform existing approaches on noisy datasets.

Abstract

Instance-dependent label noise is realistic but rather challenging, where the label-corruption process depends on instances directly. It causes a severe distribution shift between the distributions of training and test data, which impairs the generalization of trained models. Prior works put great effort into tackling the issue. Unfortunately, these works always highly rely on strong assumptions or remain heuristic without theoretical guarantees. In this paper, to address the distribution shift in learning with instance-dependent label noise, a dynamic distribution-calibration strategy is adopted. Specifically, we hypothesize that, before training data are corrupted by label noise, each class conforms to a multivariate Gaussian distribution at the feature level. Label noise produces outliers to shift the Gaussian distribution. During training, to calibrate the shifted distribution, we propose two methods based on the mean and covariance of multivariate Gaussian distribution respectively. The mean-based method works in a recursive dimension-reduction manner for robust mean estimation, which is theoretically guaranteed to train a high-quality model against label noise. The covariance-based method works in a distribution disturbance manner, which is experimentally verified to improve the model robustness. We demonstrate the utility and effectiveness of our methods on datasets with synthetic label noise and real-world unknown noise.