Regularly Truncated M-estimators for Learning with Noisy Labels

TL;DR

This paper introduces RTME, a novel approach that adaptively switches between truncated and original M-estimators to improve learning with noisy labels, effectively handling noise and utilizing all data for better generalization.

Contribution

The paper proposes RTME, a new method that alternates between truncated and original M-estimators to address noisy labels and leverage all training data.

Findings

RTME outperforms multiple baselines across various noise levels.

RTME demonstrates robustness to different types of label noise.

Theoretical analysis confirms label-noise-tolerance of RTME.

Abstract

The sample selection approach is very popular in learning with noisy labels. As deep networks learn pattern first, prior methods built on sample selection share a similar training procedure: the small-loss examples can be regarded as clean examples and used for helping generalization, while the large-loss examples are treated as mislabeled ones and excluded from network parameter updates. However, such a procedure is arguably debatable from two folds: (a) it does not consider the bad influence of noisy labels in selected small-loss examples; (b) it does not make good use of the discarded large-loss examples, which may be clean or have meaningful information for generalization. In this paper, we propose regularly truncated M-estimators (RTME) to address the above two issues simultaneously. Specifically, RTME can alternately switch modes between truncated M-estimators and original M-estimators. The former can adaptively select small-losses examples without knowing the noise rate and reduce the side-effects of noisy labels in them. The latter makes the possibly clean examples but with large losses involved to help generalization. Theoretically, we demonstrate that our strategies are label-noise-tolerant. Empirically, comprehensive experimental results show that our method can outperform multiple baselines and is robust to broad noise types and levels.