Pre-Trained Model Recommendation for Downstream Fine-tuning

TL;DR

This paper introduces Fennec, a transfer learning framework that maps models and tasks into a transfer space for efficient model selection, leveraging a large repository and a novel encoding method, archi2vec.

Contribution

The paper proposes a new transferability-based model selection framework, Fennec, with a novel model encoding method, archi2vec, and provides a comprehensive benchmark for evaluation.

Findings

Fennec effectively ranks models with minimal computation.

The transfer score computation is O(1) in time complexity.

Benchmark results validate the framework's effectiveness.

Abstract

As a fundamental problem in transfer learning, model selection aims to rank off-the-shelf pre-trained models and select the most suitable one for the new target task. Existing model selection techniques are often constrained in their scope and tend to overlook the nuanced relationships between models and tasks. In this paper, we present a pragmatic framework \textbf{Fennec}, delving into a diverse, large-scale model repository while meticulously considering the intricate connections between tasks and models. The key insight is to map all models and historical tasks into a transfer-related subspace, where the distance between model vectors and task vectors represents the magnitude of transferability. A large vision model, as a proxy, infers a new task's representation in the transfer space, thereby circumventing the computational burden of extensive forward passes. We also investigate the impact of the inherent inductive bias of models on transfer results and propose a novel method called \textbf{archi2vec} to encode the intricate structures of models. The transfer score is computed through straightforward vector arithmetic with a time complexity of $\mathcal{O}(1)$. Finally, we make a substantial contribution to the field by releasing a comprehensive benchmark. We validate the effectiveness of our framework through rigorous testing on two benchmarks. The benchmark and the code will be publicly available in the near future.