Hard Samples, Bad Labels: Robust Loss Functions That Know When to Back Off

TL;DR

This paper introduces two novel loss functions, Blurry Loss and Piecewise-zero Loss, designed to improve robustness against label errors by de-emphasizing difficult samples, thereby enhancing error detection and data cleaning in supervised learning.

Contribution

The paper proposes two new loss functions that improve robustness to label errors and outperform existing methods in error detection across various datasets.

Findings

Outperform state-of-the-art robust loss functions in error detection

Effective across both uniform and non-uniform label corruption

Enhance data cleaning by better identifying mislabeled samples

Abstract

Incorrectly labelled training data are frustratingly ubiquitous in both benchmark and specially curated datasets. Such mislabelling clearly adversely affects the performance and generalizability of models trained through supervised learning on the associated datasets. Frameworks for detecting label errors typically require well-trained / well-generalized models; however, at the same time most frameworks rely on training these models on corrupt data, which clearly has the effect of reducing model generalizability and subsequent effectiveness in error detection -- unless a training scheme robust to label errors is employed. We evaluate two novel loss functions, Blurry Loss and Piecewise-zero Loss, that enhance robustness to label errors by de-weighting or disregarding difficult-to-classify samples, which are likely to be erroneous. These loss functions leverage the idea that mislabelled examples are typically more difficult to classify and should contribute less to the learning signal. Comprehensive experiments on a variety of artificially corrupted datasets demonstrate that the proposed loss functions outperform state-of-the-art robust loss functions in nearly all cases, achieving superior F1 scores for error detection. Further analyses through ablation studies offer insights to confirm these loss functions' broad applicability to cases of both uniform and non-uniform corruption, and with different label error detection frameworks. By using these robust loss functions, machine learning practitioners can more effectively identify, prune, or correct errors in their training data.