ELM: Embedding and Logit Margins for Long-Tail Learning

TL;DR

ELM introduces a unified margin-based approach that regularizes embeddings and logits to improve long-tail learning, leading to better generalization and tighter tail class embeddings.

Contribution

The paper proposes ELM, a novel method that enforces margins in logit space and regularizes embeddings, bridging long-tail learning with metric embedding and contrastive learning.

Findings

ELM improves tail class generalization.

ELM results in tighter tail class embeddings.

Theoretical analysis shows reduced generalization gap.

Abstract

Long-tail learning is the problem of learning under skewed label distributions, which pose a challenge for standard learners. Several recent approaches for the problem have proposed enforcing a suitable margin in logit space. Such techniques are intuitive analogues of the guiding principle behind SVMs, and are equally applicable to linear models and neural models. However, when applied to neural models, such techniques do not explicitly control the geometry of the learned embeddings. This can be potentially sub-optimal, since embeddings for tail classes may be diffuse, resulting in poor generalization for these classes. We present Embedding and Logit Margins (ELM), a unified approach to enforce margins in logit space, and regularize the distribution of embeddings. This connects losses for long-tail learning to proposals in the literature on metric embedding, and contrastive learning. We theoretically show that minimising the proposed ELM objective helps reduce the generalisation gap. The ELM method is shown to perform well empirically, and results in tighter tail class embeddings.