Foundation Model Insights and a Multi-Model Approach for Superior Fine-Grained One-shot Subset Selection

TL;DR

This paper explores the use of foundation models for one-shot subset selection, demonstrating their superiority over traditional methods on fine-grained datasets and introducing RAM-APL, a multi-model approach that achieves state-of-the-art results.

Contribution

It introduces RAM-APL, a novel multi-model method leveraging foundation models to improve fine-grained subset selection performance.

Findings

FMs outperform traditional IEs on fine-grained datasets.

The advantage of FMs diminishes on coarse-grained datasets with noisy labels.

RAM-APL achieves state-of-the-art results on several fine-grained datasets.

Abstract

One-shot subset selection serves as an effective tool to reduce deep learning training costs by identifying an informative data subset based on the information extracted by an information extractor (IE). Traditional IEs, typically pre-trained on the target dataset, are inherently dataset-dependent. Foundation models (FMs) offer a promising alternative, potentially mitigating this limitation. This work investigates two key questions: (1) Can FM-based subset selection outperform traditional IE-based methods across diverse datasets? (2) Do all FMs perform equally well as IEs for subset selection? Extensive experiments uncovered surprising insights: FMs consistently outperform traditional IEs on fine-grained datasets, whereas their advantage diminishes on coarse-grained datasets with noisy labels. Motivated by these finding, we propose RAM-APL (RAnking Mean-Accuracy of Pseudo-class Labels), a method tailored for fine-grained image datasets. RAM-APL leverages multiple FMs to enhance subset selection by exploiting their complementary strengths. Our approach achieves state-of-the-art performance on fine-grained datasets, including Oxford-IIIT Pet, Food-101, and Caltech-UCSD Birds-200-2011.