Robustness and Reliability When Training With Noisy Labels

TL;DR

This paper analyzes how label noise affects the training of discriminative models, highlighting limitations in robustness and calibration, and emphasizes the need for algorithms that ensure both accuracy and reliability.

Contribution

It provides a theoretical analysis of label noise impact on model calibration and robustness, and evaluates the effectiveness of robust loss functions in noisy label scenarios.

Findings

Strictly proper and robust loss functions offer asymptotic accuracy robustness.

Neither loss type guarantees model calibration under label noise.

Early stopping helps mitigate overfitting to noisy labels.

Abstract

Labelling of data for supervised learning can be costly and time-consuming and the risk of incorporating label noise in large data sets is imminent. When training a flexible discriminative model using a strictly proper loss, such noise will inevitably shift the solution towards the conditional distribution over noisy labels. Nevertheless, while deep neural networks have proven capable of fitting random labels, regularisation and the use of robust loss functions empirically mitigate the effects of label noise. However, such observations concern robustness in accuracy, which is insufficient if reliable uncertainty quantification is critical. We demonstrate this by analysing the properties of the conditional distribution over noisy labels for an input-dependent noise model. In addition, we evaluate the set of robust loss functions characterised by noise-insensitive, asymptotic risk minimisers. We find that strictly proper and robust loss functions both offer asymptotic robustness in accuracy, but neither guarantee that the final model is calibrated. Moreover, even with robust loss functions, overfitting is an issue in practice. With these results, we aim to explain observed robustness of common training practices, such as early stopping, to label noise. In addition, we aim to encourage the development of new noise-robust algorithms that not only preserve accuracy but that also ensure reliability.