Scalable Diverse Model Selection for Accessible Transfer Learning

TL;DR

This paper introduces a new framework and benchmarks for quickly selecting the best pretrained models for transfer learning from large model banks, addressing scalability and diversity challenges.

Contribution

It formalizes the scalable diverse model selection problem, evaluates existing methods, and proposes PARC, a new approach that outperforms others in this setting.

Findings

Existing methods perform poorly on diverse model selection tasks.

Simple techniques can improve transferability estimation performance.

PARC outperforms all other methods on the proposed benchmarks.

Abstract

With the preponderance of pretrained deep learning models available off-the-shelf from model banks today, finding the best weights to fine-tune to your use-case can be a daunting task. Several methods have recently been proposed to find good models for transfer learning, but they either don't scale well to large model banks or don't perform well on the diversity of off-the-shelf models. Ideally the question we want to answer is, "given some data and a source model, can you quickly predict the model's accuracy after fine-tuning?" In this paper, we formalize this setting as "Scalable Diverse Model Selection" and propose several benchmarks for evaluating on this task. We find that existing model selection and transferability estimation methods perform poorly here and analyze why this is the case. We then introduce simple techniques to improve the performance and speed of these algorithms. Finally, we iterate on existing methods to create PARC, which outperforms all other methods on diverse model selection. We have released the benchmarks and method code in hope to inspire future work in model selection for accessible transfer learning.