A Robust Optimization Method for Label Noisy Datasets Based on Adaptive Threshold: Adaptive-k

TL;DR

Adaptive-k is a robust optimization method that dynamically excludes noisy labels during training, improving accuracy on noisy datasets without prior noise knowledge or extra training overhead.

Contribution

The paper introduces Adaptive-k, a novel threshold-based method that effectively excludes noisy samples during training, outperforming fixed or all-sample approaches without additional noise ratio information.

Findings

Adaptive-k outperforms traditional SGD on noisy datasets.

The method closely matches the performance of an oracle with perfect noise removal.

Adaptive-k does not require extra training or prior noise knowledge.

Abstract

SGD does not produce robust results on datasets with label noise. Because the gradients calculated according to the losses of the noisy samples cause the optimization process to go in the wrong direction. In this paper, as an alternative to SGD, we recommend using samples with loss less than a threshold value determined during the optimization process, instead of using all samples in the mini-batch. Our proposed method, Adaptive-k, aims to exclude label noise samples from the optimization process and make the process robust. On noisy datasets, we found that using a threshold-based approach, such as Adaptive-k, produces better results than using all samples or a fixed number of low-loss samples in the mini-batch. Based on our theoretical analysis and experimental results, we show that the Adaptive-k method is closest to the performance of the oracle, in which noisy samples are entirely removed from the dataset. Adaptive-k is a simple but effective method. It does not require prior knowledge of the noise ratio of the dataset, does not require additional model training, and does not increase training time significantly. The code for Adaptive-k is available at https://github.com/enesdedeoglu-TR/Adaptive-k