Implicit Modeling for Transferability Estimation of Vision Foundation Models

TL;DR

This paper introduces Implicit Transferability Modeling (ITM), a novel framework that accurately estimates the transferability of diverse vision models to downstream tasks efficiently, facilitating better model selection without costly fine-tuning.

Contribution

The paper proposes a new implicit transferability modeling framework with a Divide-and-Conquer Variational Approximation strategy that generalizes across various models and tasks.

Findings

ITM outperforms existing methods in stability and effectiveness.

ITM demonstrates higher efficiency in transferability estimation.

Extensive experiments validate ITM's broad applicability and superior performance.

Abstract

Transferability estimation identifies the best pre-trained models for downstream tasks without incurring the high computational cost of full fine-tuning. This capability facilitates deployment and advances the pre-training and fine-tuning paradigm. However, existing methods often struggle to accurately assess transferability for emerging pre-trained models with diverse architectures, training strategies, and task alignments. In this work, we propose Implicit Transferability Modeling (ITM), a novel framework that implicitly models each model's intrinsic transferability, coupled with a Divide-and-Conquer Variational Approximation (DVA) strategy to efficiently approximate embedding space evolution. This design enables generalization across a broader range of models and downstream tasks. Extensive experiments on a comprehensive benchmark--spanning extensive training regimes and a wider variety of model types--demonstrate that ITM consistently outperforms existing methods in terms of stability, effectiveness, and efficiency.