An Empirical Study on Clustering Pretrained Embeddings: Is Deep Strictly Better?

TL;DR

This study empirically compares various clustering methods on face embeddings, revealing that deep methods are fragile with uncertain embeddings and only marginally outperform shallow methods on highly discriminative data.

Contribution

It provides a large-scale, robust evaluation of 17 clustering methods across multiple datasets, broadening the understanding of their performance beyond face data.

Findings

Deep methods are fragile with uncertain embeddings.

Shallow methods perform comparably on less discriminative data.

Deep methods only slightly outperform shallow methods on highly discriminative embeddings.

Abstract

Recent research in clustering face embeddings has found that unsupervised, shallow, heuristic-based methods -- including $k$-means and hierarchical agglomerative clustering -- underperform supervised, deep, inductive methods. While the reported improvements are indeed impressive, experiments are mostly limited to face datasets, where the clustered embeddings are highly discriminative or well-separated by class (Recall@1 above 90% and often nearing ceiling), and the experimental methodology seemingly favors the deep methods. We conduct a large-scale empirical study of 17 clustering methods across three datasets and obtain several robust findings. Notably, deep methods are surprisingly fragile for embeddings with more uncertainty, where they match or even perform worse than shallow, heuristic-based methods. When embeddings are highly discriminative, deep methods do outperform the baselines, consistent with past results, but the margin between methods is much smaller than previously reported. We believe our benchmarks broaden the scope of supervised clustering methods beyond the face domain and can serve as a foundation on which these methods could be improved. To enable reproducibility, we include all necessary details in the appendices, and plan to release the code.