The Pursuit of Human Labeling: A New Perspective on Unsupervised Learning

TL;DR

HUME is a model-agnostic framework that infers human-like labels from datasets by leveraging the linear separability of classes across various representations, outperforming existing methods on multiple image classification benchmarks.

Contribution

This work introduces HUME, a novel unsupervised learning approach that searches for consistent labelings across representations, providing a new perspective on tackling unsupervised classification tasks.

Findings

HUME outperforms supervised linear classifiers on STL-10.

Achieves state-of-the-art results on four benchmark datasets.

Compatible with any large pretrained self-supervised model.

Abstract

We present HUME, a simple model-agnostic framework for inferring human labeling of a given dataset without any external supervision. The key insight behind our approach is that classes defined by many human labelings are linearly separable regardless of the representation space used to represent a dataset. HUME utilizes this insight to guide the search over all possible labelings of a dataset to discover an underlying human labeling. We show that the proposed optimization objective is strikingly well-correlated with the ground truth labeling of the dataset. In effect, we only train linear classifiers on top of pretrained representations that remain fixed during training, making our framework compatible with any large pretrained and self-supervised model. Despite its simplicity, HUME outperforms a supervised linear classifier on top of self-supervised representations on the STL-10 dataset by a large margin and achieves comparable performance on the CIFAR-10 dataset. Compared to the existing unsupervised baselines, HUME achieves state-of-the-art performance on four benchmark image classification datasets including the large-scale ImageNet-1000 dataset. Altogether, our work provides a fundamentally new view to tackle unsupervised learning by searching for consistent labelings between different representation spaces.