Long-tail learning via logit adjustment

TL;DR

This paper introduces two simple logit adjustment methods based on label frequencies to improve classification performance on long-tailed datasets, addressing bias towards dominant labels and enhancing generalization.

Contribution

It proposes two modifications of softmax cross-entropy using logit adjustment, unifying and generalizing recent approaches with stronger statistical basis and empirical validation.

Findings

Improved classification on long-tailed datasets.

Logit adjustment enhances margin between rare and dominant labels.

Methods outperform baseline models in experiments.

Abstract

Real-world classification problems typically exhibit an imbalanced or long-tailed label distribution, wherein many labels are associated with only a few samples. This poses a challenge for generalisation on such labels, and also makes na\"ive learning biased towards dominant labels. In this paper, we present two simple modifications of standard softmax cross-entropy training to cope with these challenges. Our techniques revisit the classic idea of logit adjustment based on the label frequencies, either applied post-hoc to a trained model, or enforced in the loss during training. Such adjustment encourages a large relative margin between logits of rare versus dominant labels. These techniques unify and generalise several recent proposals in the literature, while possessing firmer statistical grounding and empirical performance.