Newer is not always better: Rethinking transferability metrics, their peculiarities, stability and performance

TL;DR

This paper improves transferability metrics for model fine-tuning by addressing covariance estimation issues, proposing a faster, more accurate estimator, and highlighting problems in target task selection with solutions validated on extensive experiments.

Contribution

It introduces a shrinkage-based estimator for H-score, enhancing its performance and speed, and identifies issues with existing metrics in target task selection, offering corrections supported by large-scale experiments.

Findings

Shrinkage-based H-score outperforms traditional methods with up to 80% correlation improvement.

Proposed estimator is 3-10 times faster than LogME.

Identified and corrected issues in metrics for target task selection with varying labels and class imbalance.

Abstract

Fine-tuning of large pre-trained image and language models on small customized datasets has become increasingly popular for improved prediction and efficient use of limited resources. Fine-tuning requires identification of best models to transfer-learn from and quantifying transferability prevents expensive re-training on all of the candidate models/tasks pairs. In this paper, we show that the statistical problems with covariance estimation drive the poor performance of H-score -- a common baseline for newer metrics -- and propose shrinkage-based estimator. This results in up to 80% absolute gain in H-score correlation performance, making it competitive with the state-of-the-art LogME measure. Our shrinkage-based H-score is $3\times$-10$\times$ faster to compute compared to LogME. Additionally, we look into a less common setting of target (as opposed to source) task selection. We demonstrate previously overlooked problems in such settings with different number of labels, class-imbalance ratios etc. for some recent metrics e.g., NCE, LEEP that resulted in them being misrepresented as leading measures. We propose a correction and recommend measuring correlation performance against relative accuracy in such settings. We support our findings with ~164,000 (fine-tuning trials) experiments on both vision models and graph neural networks.