Constrained Instance and Class Reweighting for Robust Learning under Label Noise

TL;DR

This paper introduces a constrained reweighting method for training deep neural networks that improves robustness against label noise by assigning importance weights to instances and classes, with a theoretical foundation and practical efficiency.

Contribution

The paper presents a novel constrained optimization approach for importance reweighting that is computationally efficient and theoretically grounded, enhancing robustness to label noise in neural networks.

Findings

Significant performance improvements on benchmark datasets with label noise

The method provides a theoretical perspective on label smoothing heuristics

Efficient mini-batch updates eliminate the need for full dataset weight storage

Abstract

Deep neural networks have shown impressive performance in supervised learning, enabled by their ability to fit well to the provided training data. However, their performance is largely dependent on the quality of the training data and often degrades in the presence of noise. We propose a principled approach for tackling label noise with the aim of assigning importance weights to individual instances and class labels. Our method works by formulating a class of constrained optimization problems that yield simple closed form updates for these importance weights. The proposed optimization problems are solved per mini-batch which obviates the need of storing and updating the weights over the full dataset. Our optimization framework also provides a theoretical perspective on existing label smoothing heuristics for addressing label noise (such as label bootstrapping). We evaluate our method on several benchmark datasets and observe considerable performance gains in the presence of label noise.