Are Classes Clusters?

TL;DR

This paper evaluates whether recent sentence embedding models naturally form meaningful clusters corresponding to topic classes, revealing that unsupervised clustering can partially reconstruct class labels in real-world datasets.

Contribution

It provides an empirical analysis of four sentence embedding models' ability to cluster topic classes without supervision.

Findings

Clustering embeddings partially recovers topic classes.

Unsupervised clustering outperforms random chance.

Embedding models show potential for unsupervised topic discovery.

Abstract

Sentence embedding models aim to provide general purpose embeddings for sentences. Most of the models studied in this paper claim to perform well on STS tasks - but they do not report on their suitability for clustering. This paper looks at four recent sentence embedding models (Universal Sentence Encoder (Cer et al., 2018), Sentence-BERT (Reimers and Gurevych, 2019), LASER (Artetxe and Schwenk, 2019), and DeCLUTR (Giorgi et al., 2020)). It gives a brief overview of the ideas behind their implementations. It then investigates how well topic classes in two text classification datasets (Amazon Reviews (Ni et al., 2019) and News Category Dataset (Misra, 2018)) map to clusters in their corresponding sentence embedding space. While the performance of the resulting classification model is far from perfect, it is better than random. This is interesting because the classification model has been constructed in an unsupervised way. The topic classes in these real life topic classification datasets can be partly reconstructed by clustering the corresponding sentence embeddings.