Let Go of Your Labels with Unsupervised Transfer

TL;DR

TURTLE is a fully unsupervised method that discovers dataset labelings by maximizing margin classifiers in foundation model spaces, outperforming zero-shot transfer on diverse tasks without supervision.

Contribution

Introduces TURTLE, a novel unsupervised approach that uncovers dataset labels in representation spaces, eliminating the need for human guidance or task-specific training.

Findings

TURTLE achieves state-of-the-art unsupervised performance on 26 datasets.

TURTLE outperforms zero-shot transfer baselines across various datasets.

TURTLE matches CLIP zero-shot performance using the same representation space.

Abstract

Foundation vision-language models have enabled remarkable zero-shot transferability of the pre-trained representations to a wide range of downstream tasks. However, to solve a new task, zero-shot transfer still necessitates human guidance to define visual categories that appear in the data. Here, we show that fully unsupervised transfer emerges when searching for the labeling of a dataset that induces maximal margin classifiers in representation spaces of different foundation models. We present TURTLE, a fully unsupervised method that effectively employs this guiding principle to uncover the underlying labeling of a downstream dataset without any supervision and task-specific representation learning. We evaluate TURTLE on a diverse benchmark suite of 26 datasets and show that it achieves new state-of-the-art unsupervised performance. Furthermore, TURTLE, although being fully unsupervised, outperforms zero-shot transfer baselines on a wide range of datasets. In particular, TURTLE matches the average performance of CLIP zero-shot on 26 datasets by employing the same representation space, spanning a wide range of architectures and model sizes. By guiding the search for the underlying labeling using the representation spaces of two foundation models, TURTLE surpasses zero-shot transfer and unsupervised prompt tuning baselines, demonstrating the surprising power and effectiveness of unsupervised transfer.