Do We Need to Penalize Variance of Losses for Learning with Label Noise?

TL;DR

This paper challenges the conventional wisdom by showing that increasing loss variance, rather than penalizing it, can improve learning with noisy labels by enhancing memorization and reducing label noise harm.

Contribution

It introduces a novel approach that increases loss variance to better handle label noise, supported by theoretical insights and empirical validation.

Findings

Increasing loss variance boosts memorization effects.

Reducing loss variance can harm learning with noisy labels.

Proposed method improves baseline performance on multiple datasets.

Abstract

Algorithms which minimize the averaged loss have been widely designed for dealing with noisy labels. Intuitively, when there is a finite training sample, penalizing the variance of losses will improve the stability and generalization of the algorithms. Interestingly, we found that the variance should be increased for the problem of learning with noisy labels. Specifically, increasing the variance will boost the memorization effects and reduce the harmfulness of incorrect labels. By exploiting the label noise transition matrix, regularizers can be easily designed to reduce the variance of losses and be plugged in many existing algorithms. Empirically, the proposed method by increasing the variance of losses significantly improves the generalization ability of baselines on both synthetic and real-world datasets.