Transferability Estimation using Bhattacharyya Class Separability

TL;DR

This paper introduces GBC, a new method to estimate transferability of pre-trained models to target tasks by measuring class separability in feature space, enabling better model and dataset selection.

Contribution

The paper proposes Gaussian Bhattacharyya Coefficient (GBC), a novel, efficient transferability metric applicable to classification and segmentation tasks, outperforming existing methods in various settings.

Findings

GBC outperforms state-of-the-art metrics in semantic segmentation transferability.

GBC matches top methods for dataset transferability in image classification.

GBC excels in architecture selection for image classification.

Abstract

Transfer learning has become a popular method for leveraging pre-trained models in computer vision. However, without performing computationally expensive fine-tuning, it is difficult to quantify which pre-trained source models are suitable for a specific target task, or, conversely, to which tasks a pre-trained source model can be easily adapted to. In this work, we propose Gaussian Bhattacharyya Coefficient (GBC), a novel method for quantifying transferability between a source model and a target dataset. In a first step we embed all target images in the feature space defined by the source model, and represent them with per-class Gaussians. Then, we estimate their pairwise class separability using the Bhattacharyya coefficient, yielding a simple and effective measure of how well the source model transfers to the target task. We evaluate GBC on image classification tasks in the context of dataset and architecture selection. Further, we also perform experiments on the more complex semantic segmentation transferability estimation task. We demonstrate that GBC outperforms state-of-the-art transferability metrics on most evaluation criteria in the semantic segmentation settings, matches the performance of top methods for dataset transferability in image classification, and performs best on architecture selection problems for image classification.