PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks

TL;DR

PACTran introduces theoretically grounded PAC-Bayesian metrics for selecting pretrained models, demonstrating superior transferability measurement across vision and language tasks compared to existing heuristics.

Contribution

The paper develops PACTran, a novel family of transferability metrics derived from PAC-Bayesian bounds, providing a theoretically justified approach for pretrained model selection.

Findings

PACTran outperforms existing heuristics in transferability measurement.

Empirical evaluation on VTAB and OKVQA shows PACTran's effectiveness.

PACTran offers a more consistent transferability assessment.

Abstract

With the increasing abundance of pretrained models in recent years, the problem of selecting the best pretrained checkpoint for a particular downstream classification task has been gaining increased attention. Although several methods have recently been proposed to tackle the selection problem (e.g. LEEP, H-score), these methods resort to applying heuristics that are not well motivated by learning theory. In this paper we present PACTran, a theoretically grounded family of metrics for pretrained model selection and transferability measurement. We first show how to derive PACTran metrics from the optimal PAC-Bayesian bound under the transfer learning setting. We then empirically evaluate three metric instantiations of PACTran on a number of vision tasks (VTAB) as well as a language-and-vision (OKVQA) task. An analysis of the results shows PACTran is a more consistent and effective transferability measure compared to existing selection methods.