Label Noise: Correcting the Forward-Correction

TL;DR

This paper introduces a method to improve neural network robustness against label noise by imposing a loss lower bound, supported by theoretical analysis and empirical validation, to prevent overfitting to noisy labels.

Contribution

It provides a novel theoretical framework for bounding the training loss based on estimated noise rates to combat overfitting in noisy datasets.

Findings

Significantly improves robustness to label noise

Requires minimal additional computational cost

Effective across various noisy data scenarios

Abstract

Training neural network classifiers on datasets with label noise poses a risk of overfitting them to the noisy labels. To address this issue, researchers have explored alternative loss functions that aim to be more robust. The `forward-correction' is a popular approach wherein the model outputs are noised before being evaluated against noisy data. When the true noise model is known, applying the forward-correction guarantees consistency of the learning algorithm. While providing some benefit, the correction is insufficient to prevent overfitting to finite noisy datasets. In this work, we propose an approach to tackling overfitting caused by label noise. We observe that the presence of label noise implies a lower bound on the noisy generalised risk. Motivated by this observation, we propose imposing a lower bound on the training loss to mitigate overfitting. Our main contribution is providing theoretical insights that allow us to approximate the lower bound given only an estimate of the average noise rate. We empirically demonstrate that using this bound significantly enhances robustness in various settings, with virtually no additional computational cost.