Tune It or Don't Use It: Benchmarking Data-Efficient Image Classification

TL;DR

This paper introduces a comprehensive benchmark for data-efficient image classification across diverse datasets and demonstrates that careful hyper-parameter tuning of standard models can outperform many specialized methods.

Contribution

It provides a standardized benchmark for fair comparison and shows that tuning basic hyper-parameters can rival or surpass recent specialized approaches.

Findings

Tuned baseline models outperform most specialized methods.

Hyper-parameter tuning is crucial for fair evaluation.

A diverse benchmark dataset enhances evaluation robustness.

Abstract

Data-efficient image classification using deep neural networks in settings, where only small amounts of labeled data are available, has been an active research area in the recent past. However, an objective comparison between published methods is difficult, since existing works use different datasets for evaluation and often compare against untuned baselines with default hyper-parameters. We design a benchmark for data-efficient image classification consisting of six diverse datasets spanning various domains (e.g., natural images, medical imagery, satellite data) and data types (RGB, grayscale, multispectral). Using this benchmark, we re-evaluate the standard cross-entropy baseline and eight methods for data-efficient deep learning published between 2017 and 2021 at renowned venues. For a fair and realistic comparison, we carefully tune the hyper-parameters of all methods on each dataset. Surprisingly, we find that tuning learning rate, weight decay, and batch size on a separate validation split results in a highly competitive baseline, which outperforms all but one specialized method and performs competitively to the remaining one.