Space Alignment Matters: The Missing Piece for Inducing Neural Collapse in Long-Tailed Learning

TL;DR

This paper highlights the importance of space alignment in neural collapse for long-tailed learning, proposing strategies to improve class feature and classifier alignment, leading to state-of-the-art results.

Contribution

It identifies the critical misalignment issue between feature and classifier spaces in long-tailed regimes and introduces three alignment strategies that enhance existing methods.

Findings

Improved class separation and generalization in long-tailed datasets

Achieved state-of-the-art performance on CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT

Theoretical analysis quantifies the harm of feature-classifier misalignment.

Abstract

Recent studies on Neural Collapse (NC) reveal that, under class-balanced conditions, the class feature means and classifier weights spontaneously align into a simplex equiangular tight frame (ETF). In long-tailed regimes, however, severe sample imbalance tends to prevent the emergence of the NC phenomenon, resulting in poor generalization performance. Current efforts predominantly seek to recover the ETF geometry by imposing constraints on features or classifier weights, yet overlook a critical problem: There is a pronounced misalignment between the feature and the classifier weight spaces. In this paper, we theoretically quantify the harm of such misalignment through an optimal error exponent analysis. Built on this insight, we propose three explicit alignment strategies that plug-and-play into existing long-tail methods without architectural change. Extensive experiments on the CIFAR-10-LT, CIFAR-100-LT, and ImageNet-LT datasets consistently boost examined baselines and achieve the state-of-the-art performances.