Unleashing the power of Neural Collapse for Transferability Estimation

TL;DR

This paper introduces FaCe, a transferability estimation method based on Neural Collapse, which effectively predicts the suitability of pre-trained models for various downstream tasks by measuring class separation, fairness, and neural collapse.

Contribution

The paper proposes FaCe, a novel neural collapse-based metric for transferability estimation, demonstrating state-of-the-art performance across multiple tasks and architectures.

Findings

FaCe correlates strongly with transfer performance.

FaCe outperforms existing transferability metrics.

The method generalizes across tasks and architectures.

Abstract

Transferability estimation aims to provide heuristics for quantifying how suitable a pre-trained model is for a specific downstream task, without fine-tuning them all. Prior studies have revealed that well-trained models exhibit the phenomenon of Neural Collapse. Based on a widely used neural collapse metric in existing literature, we observe a strong correlation between the neural collapse of pre-trained models and their corresponding fine-tuned models. Inspired by this observation, we propose a novel method termed Fair Collapse (FaCe) for transferability estimation by comprehensively measuring the degree of neural collapse in the pre-trained model. Typically, FaCe comprises two different terms: the variance collapse term, which assesses the class separation and within-class compactness, and the class fairness term, which quantifies the fairness of the pre-trained model towards each class. We investigate FaCe on a variety of pre-trained classification models across different network architectures, source datasets, and training loss functions. Results show that FaCe yields state-of-the-art performance on different tasks including image classification, semantic segmentation, and text classification, which demonstrate the effectiveness and generalization of our method.