A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark

TL;DR

This paper introduces the Visual Task Adaptation Benchmark (VTAB) to evaluate the generalization of visual representations across diverse tasks, providing insights into the effectiveness of various learning algorithms and supervision methods.

Contribution

The paper presents VTAB, a comprehensive benchmark for assessing visual representations on diverse tasks, and conducts a large-scale study comparing different learning algorithms and supervision techniques.

Findings

ImageNet representations perform well beyond natural datasets

Generative and discriminative models show comparable effectiveness

Self-supervision can often replace labels effectively

Abstract

Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?