Dynamic Loss For Robust Learning

TL;DR

This paper introduces a meta-learning based dynamic loss function that adaptively corrects noisy labels and adjusts class margins to improve classifier robustness on long-tailed noisy datasets, achieving state-of-the-art results.

Contribution

It proposes a novel dynamic loss with label correction and margin generation components, optimized via meta-learning, to handle combined label noise and class imbalance.

Findings

Achieves state-of-the-art accuracy on multiple datasets

Effectively handles noisy and long-tailed data

Demonstrates robustness across real-world and synthetic datasets

Abstract

Label noise and class imbalance commonly coexist in real-world data. Previous works for robust learning, however, usually address either one type of the data biases and underperform when facing them both. To mitigate this gap, this work presents a novel meta-learning based dynamic loss that automatically adjusts the objective functions with the training process to robustly learn a classifier from long-tailed noisy data. Concretely, our dynamic loss comprises a label corrector and a margin generator, which respectively correct noisy labels and generate additive per-class classification margins by perceiving the underlying data distribution as well as the learning state of the classifier. Equipped with a new hierarchical sampling strategy that enriches a small amount of unbiased metadata with diverse and hard samples, the two components in the dynamic loss are optimized jointly through meta-learning and cultivate the classifier to well adapt to clean and balanced test data. Extensive experiments show our method achieves state-of-the-art accuracy on multiple real-world and synthetic datasets with various types of data biases, including CIFAR-10/100, Animal-10N, ImageNet-LT, and Webvision. Code will soon be publicly available.