Not All Models Are Equal: Predicting Model Transferability in a Self-challenging Fisher Space

TL;DR

This paper introduces SFDA, a novel transferability metric that predicts the effectiveness of pre-trained models for downstream tasks by embedding features into Fisher space and applying a self-challenging mechanism, outperforming existing methods.

Contribution

The paper proposes SFDA, a new transferability metric that captures fine-tuning dynamics and improves model ranking accuracy with efficiency and robustness.

Findings

SFDA outperforms state-of-the-art methods like NLEEP by 59.1% in transferability prediction.

SFDA achieves 22.5x speedup in wall-clock time compared to previous methods.

Extensive experiments on 33 models and 11 tasks validate SFDA's effectiveness and robustness.

Abstract

This paper addresses an important problem of ranking the pre-trained deep neural networks and screening the most transferable ones for downstream tasks. It is challenging because the ground-truth model ranking for each task can only be generated by fine-tuning the pre-trained models on the target dataset, which is brute-force and computationally expensive. Recent advanced methods proposed several lightweight transferability metrics to predict the fine-tuning results. However, these approaches only capture static representations but neglect the fine-tuning dynamics. To this end, this paper proposes a new transferability metric, called \textbf{S}elf-challenging \textbf{F}isher \textbf{D}iscriminant \textbf{A}nalysis (\textbf{SFDA}), which has many appealing benefits that existing works do not have. First, SFDA can embed the static features into a Fisher space and refine them for better separability between classes. Second, SFDA uses a self-challenging mechanism to encourage different pre-trained models to differentiate on hard examples. Third, SFDA can easily select multiple pre-trained models for the model ensemble. Extensive experiments on $33$ pre-trained models of $11$ downstream tasks show that SFDA is efficient, effective, and robust when measuring the transferability of pre-trained models. For instance, compared with the state-of-the-art method NLEEP, SFDA demonstrates an average of $59.1$\% gain while bringing $22.5$x speedup in wall-clock time. The code will be available at \url{https://github.com/TencentARC/SFDA}.