Where Should I Spend My FLOPS? Efficiency Evaluations of Visual Pre-training Methods

TL;DR

This paper evaluates the efficiency of various visual pre-training methods and datasets under fixed FLOP budgets, highlighting disparities in computational efficiency and emphasizing the importance of dataset quality and cost-aware reporting.

Contribution

It provides a comprehensive analysis of pre-training methods and datasets, comparing their FLOP and CO2 footprints relative to accuracy on visual tasks, challenging assumptions about self-supervised scaling.

Findings

Self-supervised methods vary greatly in computational efficiency.

Dataset quality significantly impacts pre-training effectiveness.

Cost-aware evaluation is crucial for fair comparison.

Abstract

Self-supervised methods have achieved remarkable success in transfer learning, often achieving the same or better accuracy than supervised pre-training. Most prior work has done so by increasing pre-training computation by adding complex data augmentation, multiple views, or lengthy training schedules. In this work, we investigate a related, but orthogonal question: given a fixed FLOP budget, what are the best datasets, models, and (self-)supervised training methods for obtaining high accuracy on representative visual tasks? Given the availability of large datasets, this setting is often more relevant for both academic and industry labs alike. We examine five large-scale datasets (JFT-300M, ALIGN, ImageNet-1K, ImageNet-21K, and COCO) and six pre-training methods (CLIP, DINO, SimCLR, BYOL, Masked Autoencoding, and supervised). In a like-for-like fashion, we characterize their FLOP and CO$_2$ footprints, relative to their accuracy when transferred to a canonical image segmentation task. Our analysis reveals strong disparities in the computational efficiency of pre-training methods and their dependence on dataset quality. In particular, our results call into question the commonly-held assumption that self-supervised methods inherently scale to large, uncurated data. We therefore advocate for (1) paying closer attention to dataset curation and (2) reporting of accuracies in context of the total computational cost.