Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal

TL;DR

This paper introduces AGRA, a dynamic method for handling noisy labels by adaptively removing outliers during training based on gradient comparisons, improving model robustness without permanently discarding data.

Contribution

The paper presents AGRA, a novel adaptive gradient-based outlier removal technique that adjusts the dataset during training, unlike previous static outlier removal methods.

Findings

AGRA outperforms traditional methods on multiple datasets.

Dynamic outlier removal benefits model training more than permanent filtering.

Extensive experiments validate AGRA's effectiveness.

Abstract

An accurate and substantial dataset is essential for training a reliable and well-performing model. However, even manually annotated datasets contain label errors, not to mention automatically labeled ones. Previous methods for label denoising have primarily focused on detecting outliers and their permanent removal - a process that is likely to over- or underfilter the dataset. In this work, we propose AGRA: a new method for learning with noisy labels by using Adaptive GRAdient-based outlier removal. Instead of cleaning the dataset prior to model training, the dataset is dynamically adjusted during the training process. By comparing the aggregated gradient of a batch of samples and an individual example gradient, our method dynamically decides whether a corresponding example is helpful for the model at this point or is counter-productive and should be left out for the current update. Extensive evaluation on several datasets demonstrates AGRA's effectiveness, while a comprehensive results analysis supports our initial hypothesis: permanent hard outlier removal is not always what model benefits the most from.