MetaRank: Task-Aware Metric Selection for Model Transferability Estimation

TL;DR

MetaRank is a meta-learning framework that automatically selects the most effective model transferability estimation metric for a given task by encoding dataset and metric descriptions into a shared semantic space, improving transfer learning efficiency.

Contribution

It introduces a task-aware, meta-learning approach for automatic MTE metric selection using textual descriptions and a learning-to-rank formulation, addressing the task-dependency issue.

Findings

MetaRank outperforms baseline methods in ranking transferability metrics.

The approach effectively generalizes across diverse datasets and models.

It reduces the need for ad hoc metric selection in transfer learning.

Abstract

Selecting an appropriate pre-trained source model is a critical, yet computationally expensive, task in transfer learning. Model Transferability Estimation (MTE) methods address this by providing efficient proxy metrics to rank models without full fine-tuning. In practice, the choice of which MTE metric to use is often ad hoc or guided simply by a metric's average historical performance. However, we observe that the effectiveness of MTE metrics is highly task-dependent and no single metric is universally optimal across all target datasets. To address this gap, we introduce MetaRank, a meta-learning framework for automatic, task-aware MTE metric selection. We formulate metric selection as a learning-to-rank problem. Rather than relying on conventional meta-features, MetaRank encodes textual descriptions of both datasets and MTE metrics using a pretrained language model, embedding them into a shared semantic space. A meta-predictor is then trained offline on diverse meta-tasks to learn the intricate relationship between dataset characteristics and metric mechanisms, optimized with a listwise objective that prioritizes correctly ranking the top-performing metrics. During the subsequent online phase, MetaRank efficiently ranks the candidate MTE metrics for a new, unseen target dataset based on its textual description, enabling practitioners to select the most appropriate metric a priori. Extensive experiments across 11 pretrained models and 11 target datasets demonstrate the strong effectiveness of our approach.