The Impact of Geometric Complexity on Neural Collapse in Transfer Learning

TL;DR

This paper investigates how the geometric complexity of learned representations influences neural collapse in transfer learning, providing both theoretical insights and experimental evidence to explain transfer learning success.

Contribution

It introduces the concept that geometric complexity is a key factor affecting neural collapse and demonstrates its impact on transfer learning performance, especially in few-shot scenarios.

Findings

Geometric complexity affects neural collapse in pre-trained models.

Influence of geometric complexity extends to new classes in transfer learning.

Higher geometric complexity can improve downstream task performance.

Abstract

Many of the recent remarkable advances in computer vision and language models can be attributed to the success of transfer learning via the pre-training of large foundation models. However, a theoretical framework which explains this empirical success is incomplete and remains an active area of research. Flatness of the loss surface and neural collapse have recently emerged as useful pre-training metrics which shed light on the implicit biases underlying pre-training. In this paper, we explore the geometric complexity of a model's learned representations as a fundamental mechanism that relates these two concepts. We show through experiments and theory that mechanisms which affect the geometric complexity of the pre-trained network also influence the neural collapse. Furthermore, we show how this effect of the geometric complexity generalizes to the neural collapse of new classes as well, thus encouraging better performance on downstream tasks, particularly in the few-shot setting.