Vision Backbone Efficient Selection for Image Classification in Low-Data Regimes

TL;DR

This paper introduces VIBES, a method for efficiently selecting dataset-specific backbones for low-data image classification, demonstrating significant performance gains over generic benchmarks with minimal search time.

Contribution

The paper formalizes the VIBES problem, proposes heuristics for backbone selection, and validates its effectiveness across multiple datasets in low-data regimes.

Findings

VIBES can identify high-performing backbones within 10 minutes.

Simple heuristics outperform generic benchmarks in low-data classification.

Backbone effectiveness varies significantly across datasets in low-data settings.

Abstract

Transfer learning has become an essential tool in modern computer vision, allowing practitioners to leverage backbones, pretrained on large datasets, to train successful models from limited annotated data. Choosing the right backbone is crucial, especially for small datasets, since final performance depends heavily on the quality of the initial feature representations. While prior work has conducted benchmarks across various datasets to identify universal top-performing backbones, we demonstrate that backbone effectiveness is highly dataset-dependent, especially in low-data scenarios where no single backbone consistently excels. To overcome this limitation, we introduce dataset-specific backbone selection as a new research direction and investigate its practical viability in low-data regimes. Since exhaustive evaluation is computationally impractical for large backbone pools, we formalize Vision Backbone Efficient Selection (VIBES) as the problem of searching for high-performing backbones under computational constraints. We define the solution space, propose several heuristics, and demonstrate VIBES feasibility for low-data image classification by performing experiments on four diverse datasets. Our results show that even simple search strategies can find well-suited backbones within a pool of over $1300$ pretrained models, outperforming generic benchmark recommendations within just ten minutes of search time on a single GPU (NVIDIA RTX A5000).