Directional Neural Collapse Explains Few-Shot Transfer in Self-Supervised Learning

TL;DR

This paper introduces the concept of directional neural collapse (CDNV) to explain why self-supervised learning representations transfer well with few labels, linking geometric properties to generalization and multitask performance.

Contribution

It provides theoretical bounds based on directional CDNV, links decision-axis collapse to multitask orthogonality, and empirically demonstrates the phenomenon across SSL objectives and synthetic data.

Findings

Directional CDNV collapses during pretraining even when classical CDNV remains large.

Small directional CDNV leads to nearly orthogonal decision axes across tasks.

Bounds closely track few-shot error at practical shot sizes.

Abstract

Frozen self-supervised representations often transfer well with only a few labels across many semantic tasks. We argue that a single geometric quantity, \emph{directional} CDNV (decision-axis variance), sits at the core of two favorable behaviors: strong few-shot transfer within a task, and low interference across many tasks. We show that both emerge when variability \emph{along} class-separating directions is small. First, we prove sharp non-asymptotic multiclass generalization bounds for downstream classification whose leading term is the directional CDNV. The bounds include finite-shot corrections that cleanly separate intrinsic decision-axis variability from centroid-estimation error. Second, we link decision-axis collapse to multitask geometry: for independent balanced labelings, small directional CDNV across tasks forces the corresponding decision axes to be nearly orthogonal, helping a single representation support many tasks with minimal interference. Empirically, across SSL objectives, directional CDNV collapses during pretraining even when classical CDNV remains large, and our bounds closely track few-shot error at practical shot sizes. Additionally, on synthetic multitask data, we verify that SSL learns representations whose induced decision axes are nearly orthogonal. The code and project page of the paper are available at [\href{https://dlfundamentals.github.io/directional-neural-collapse/}{project page}].