Model Spider: Learning to Rank Pre-Trained Models Efficiently

TL;DR

Model Spider introduces an efficient method for selecting the most suitable pre-trained models for a given task by summarizing models and tasks into tokens and learning to rank models based on their fitness scores, reducing computational costs.

Contribution

It proposes a novel tokenization and ranking framework for efficient PTM selection that generalizes to new tasks and improves selection accuracy.

Findings

Achieves effective PTM ranking with reduced computation.

Generalizes well to unseen tasks.

Demonstrates promising results across various model zoos.

Abstract

Figuring out which Pre-Trained Model (PTM) from a model zoo fits the target task is essential to take advantage of plentiful model resources. With the availability of numerous heterogeneous PTMs from diverse fields, efficiently selecting the most suitable PTM is challenging due to the time-consuming costs of carrying out forward or backward passes over all PTMs. In this paper, we propose Model Spider, which tokenizes both PTMs and tasks by summarizing their characteristics into vectors to enable efficient PTM selection. By leveraging the approximated performance of PTMs on a separate set of training tasks, Model Spider learns to construct tokens and measure the fitness score between a model-task pair via their tokens. The ability to rank relevant PTMs higher than others generalizes to new tasks. With the top-ranked PTM candidates, we further learn to enrich task tokens with their PTM-specific semantics to re-rank the PTMs for better selection. Model Spider balances efficiency and selection ability, making PTM selection like a spider preying on a web. Model Spider demonstrates promising performance in various configurations of model zoos.