Mitigating Memorization of Noisy Labels by Clipping the Model Prediction

TL;DR

This paper introduces LogitClip, a method that clips model logits to bound the loss, thereby improving neural network robustness against noisy labels and enhancing generalization.

Contribution

It proposes a universal logit clipping technique that bounds the loss at the logit level, improving robustness of existing loss functions to noisy labels.

Findings

LogitClip significantly improves noise robustness of cross entropy loss.

LogitClip enhances generalization performance of robust losses.

Theoretical analysis confirms noise-tolerant properties of LogitClip.

Abstract

In the presence of noisy labels, designing robust loss functions is critical for securing the generalization performance of deep neural networks. Cross Entropy (CE) loss has been shown to be not robust to noisy labels due to its unboundedness. To alleviate this issue, existing works typically design specialized robust losses with the symmetric condition, which usually lead to the underfitting issue. In this paper, our key idea is to induce a loss bound at the logit level, thus universally enhancing the noise robustness of existing losses. Specifically, we propose logit clipping (LogitClip), which clamps the norm of the logit vector to ensure that it is upper bounded by a constant. In this manner, CE loss equipped with our LogitClip method is effectively bounded, mitigating the overfitting to examples with noisy labels. Moreover, we present theoretical analyses to certify the noise-tolerant ability of LogitClip. Extensive experiments show that LogitClip not only significantly improves the noise robustness of CE loss, but also broadly enhances the generalization performance of popular robust losses.