Assessing Pre-Trained Models for Transfer Learning Through Distribution of Spectral Components

TL;DR

This paper introduces DISCO, a spectral component distribution-based assessment method for pre-trained models, which predicts transferability by analyzing the spectral features' singular values, improving model selection for downstream tasks.

Contribution

It proposes a novel spectral distribution approach for evaluating pre-trained models' transferability, incorporating downstream labels for more accurate assessment.

Findings

Achieves state-of-the-art model selection performance.

Effective across classification and regression tasks.

Demonstrates robustness on multiple benchmarks.

Abstract

Pre-trained model assessment for transfer learning aims to identify the optimal candidate for the downstream tasks from a model hub, without the need of time-consuming fine-tuning. Existing advanced works mainly focus on analyzing the intrinsic characteristics of the entire features extracted by each pre-trained model or how well such features fit the target labels. This paper proposes a novel perspective for pre-trained model assessment through the Distribution of Spectral Components (DISCO). Through singular value decomposition of features extracted from pre-trained models, we investigate different spectral components and observe that they possess distinct transferability, contributing diversely to the fine-tuning performance. Inspired by this, we propose an assessment method based on the distribution of spectral components which measures the proportions of their corresponding singular values. Pre-trained models with features concentrating on more transferable components are regarded as better choices for transfer learning. We further leverage the labels of downstream data to better estimate the transferability of each spectral component and derive the final assessment criterion. Our proposed method is flexible and can be applied to both classification and regression tasks. We conducted comprehensive experiments across three benchmarks and two tasks including image classification and object detection, demonstrating that our method achieves state-of-the-art performance in choosing proper pre-trained models from the model hub for transfer learning.