Generalized Data Weighting via Class-level Gradient Manipulation

TL;DR

This paper introduces Generalized Data Weighting (GDW), a novel method that manipulates class-level gradients to simultaneously address label noise and class imbalance without extra computational cost.

Contribution

GDW unrolls loss gradients to class-level gradients and reweights them, improving performance on noisy and imbalanced datasets with efficient class-level weighting.

Findings

GDW outperforms state-of-the-art methods by 2.56% on CIFAR10 with 60% uniform noise.

GDW effectively mitigates label noise and class imbalance simultaneously.

GDW achieves these improvements without additional computational overhead.

Abstract

Label noise and class imbalance are two major issues coexisting in real-world datasets. To alleviate the two issues, state-of-the-art methods reweight each instance by leveraging a small amount of clean and unbiased data. Yet, these methods overlook class-level information within each instance, which can be further utilized to improve performance. To this end, in this paper, we propose Generalized Data Weighting (GDW) to simultaneously mitigate label noise and class imbalance by manipulating gradients at the class level. To be specific, GDW unrolls the loss gradient to class-level gradients by the chain rule and reweights the flow of each gradient separately. In this way, GDW achieves remarkable performance improvement on both issues. Aside from the performance gain, GDW efficiently obtains class-level weights without introducing any extra computational cost compared with instance weighting methods. Specifically, GDW performs a gradient descent step on class-level weights, which only relies on intermediate gradients. Extensive experiments in various settings verify the effectiveness of GDW. For example, GDW outperforms state-of-the-art methods by $2.56\%$ under the $60\%$ uniform noise setting in CIFAR10. Our code is available at https://github.com/GGchen1997/GDW-NIPS2021.