Occam's model: Selecting simpler representations for better transferability estimation

TL;DR

This paper introduces two novel metrics for transferability estimation of pre-trained models, demonstrating improved accuracy and robustness across diverse tasks, supported by theoretical insights and empirical evaluations.

Contribution

The paper proposes new transferability metrics based on model representations, with theoretical analysis and superior empirical performance over existing methods.

Findings

Metrics increase Kendall's Tau by up to 32%

Demonstrates robustness across diverse problem settings

Provides theoretical insights into transferability estimation

Abstract

Fine-tuning models that have been pre-trained on large datasets has become a cornerstone of modern machine learning workflows. With the widespread availability of online model repositories, such as Hugging Face, it is now easier than ever to fine-tune pre-trained models for specific tasks. This raises a critical question: which pre-trained model is most suitable for a given task? This problem is called transferability estimation. In this work, we introduce two novel and effective metrics for estimating the transferability of pre-trained models. Our approach is grounded in viewing transferability as a measure of how easily a pre-trained model's representations can be trained to separate target classes, providing a unique perspective on transferability estimation. We rigorously evaluate the proposed metrics against state-of-the-art alternatives across diverse problem settings, demonstrating their robustness and practical utility. Additionally, we present theoretical insights that explain our metrics' efficacy and adaptability to various scenarios. We experimentally show that our metrics increase Kendall's Tau by up to 32% compared to the state-of-the-art baselines.