$\epsilon$-Softmax: Approximating One-Hot Vectors for Mitigating Label Noise

TL;DR

This paper introduces $\\epsilon$-softmax, a method that approximates one-hot vectors to improve deep neural network robustness against label noise by relaxing the symmetric condition, with theoretical guarantees and empirical validation.

Contribution

Proposes $\\epsilon$-softmax to relax the symmetric condition, enabling noise-tolerant learning with theoretical risk bounds and improved robustness in practice.

Findings

Outperforms existing methods on synthetic label noise datasets

Achieves better robustness-accuracy trade-off on real-world noisy data

Theoretically guarantees noise tolerance with controllable excess risk

Abstract

Noisy labels pose a common challenge for training accurate deep neural networks. To mitigate label noise, prior studies have proposed various robust loss functions to achieve noise tolerance in the presence of label noise, particularly symmetric losses. However, they usually suffer from the underfitting issue due to the overly strict symmetric condition. In this work, we propose a simple yet effective approach for relaxing the symmetric condition, namely $\epsilon$-softmax, which simply modifies the outputs of the softmax layer to approximate one-hot vectors with a controllable error $\epsilon$. Essentially, $\epsilon$-softmax not only acts as an alternative for the softmax layer, but also implicitly plays the crucial role in modifying the loss function. We prove theoretically that $\epsilon$-softmax can achieve noise-tolerant learning with controllable excess risk bound for almost any loss function. Recognizing that $\epsilon$-softmax-enhanced losses may slightly reduce fitting ability on clean datasets, we further incorporate them with one symmetric loss, thereby achieving a better trade-off between robustness and effective learning. Extensive experiments demonstrate the superiority of our method in mitigating synthetic and real-world label noise. The code is available at https://github.com/cswjl/eps-softmax.