Beyond Supervised vs. Unsupervised: Representative Benchmarking and Analysis of Image Representation Learning

TL;DR

This paper critically evaluates various unsupervised image representation learning methods using multiple benchmarks and novel metrics, revealing no clear leader and emphasizing the importance of combining methods for better performance.

Contribution

It introduces new metrics for analyzing embeddings, compares multiple unsupervised methods beyond supervised vs. unsupervised, and provides insights into their complementary strengths.

Findings

No single unsupervised method dominates across benchmarks.

Different methods exhibit complementary strengths when combined.

New metrics help quantify invariance and similarity in embeddings.

Abstract

By leveraging contrastive learning, clustering, and other pretext tasks, unsupervised methods for learning image representations have reached impressive results on standard benchmarks. The result has been a crowded field - many methods with substantially different implementations yield results that seem nearly identical on popular benchmarks, such as linear evaluation on ImageNet. However, a single result does not tell the whole story. In this paper, we compare methods using performance-based benchmarks such as linear evaluation, nearest neighbor classification, and clustering for several different datasets, demonstrating the lack of a clear front-runner within the current state-of-the-art. In contrast to prior work that performs only supervised vs. unsupervised comparison, we compare several different unsupervised methods against each other. To enrich this comparison, we analyze embeddings with measurements such as uniformity, tolerance, and centered kernel alignment (CKA), and propose two new metrics of our own: nearest neighbor graph similarity and linear prediction overlap. We reveal through our analysis that in isolation, single popular methods should not be treated as though they represent the field as a whole, and that future work ought to consider how to leverage the complimentary nature of these methods. We also leverage CKA to provide a framework to robustly quantify augmentation invariance, and provide a reminder that certain types of invariance will be undesirable for downstream tasks.