No One Left Behind: Improving the Worst Categories in Long-Tailed Learning

TL;DR

This paper introduces a simple re-training method with a new loss function to improve the worst-case recall in long-tailed learning, leading to more balanced class performance and higher harmonic mean accuracy.

Contribution

It proposes a plug-in re-training approach with a novel loss function to enhance the minimum recall across categories in long-tailed datasets.

Findings

Improves the uniformity of recall across categories.

Increases harmonic mean accuracy without sacrificing average accuracy.

Effective on widely used benchmark datasets.

Abstract

Unlike the case when using a balanced training dataset, the per-class recall (i.e., accuracy) of neural networks trained with an imbalanced dataset are known to vary a lot from category to category. The convention in long-tailed recognition is to manually split all categories into three subsets and report the average accuracy within each subset. We argue that under such an evaluation setting, some categories are inevitably sacrificed. On one hand, focusing on the average accuracy on a balanced test set incurs little penalty even if some worst performing categories have zero accuracy. On the other hand, classes in the "Few" subset do not necessarily perform worse than those in the "Many" or "Medium" subsets. We therefore advocate to focus more on improving the lowest recall among all categories and the harmonic mean of all recall values. Specifically, we propose a simple plug-in method that is applicable to a wide range of methods. By simply re-training the classifier of an existing pre-trained model with our proposed loss function and using an optional ensemble trick that combines the predictions of the two classifiers, we achieve a more uniform distribution of recall values across categories, which leads to a higher harmonic mean accuracy while the (arithmetic) average accuracy is still high. The effectiveness of our method is justified on widely used benchmark datasets.