Hubness Reduction Improves Sentence-BERT Semantic Spaces

TL;DR

This paper demonstrates that reducing hubness in Sentence-BERT semantic spaces significantly improves the quality of text embeddings, leading to better performance in semantic tasks.

Contribution

The study identifies hubness as a key issue in Sentence-BERT embeddings and shows that applying combined hubness reduction methods enhances semantic space quality.

Findings

Hubness causes asymmetric neighborhood relations in embeddings.

Applying hubness reduction decreases error rates in neighborhood-based classifiers.

Combined hubness reduction methods can reduce hubness by about 75%.

Abstract

Semantic representations of text, i.e. representations of natural language which capture meaning by geometry, are essential for areas such as information retrieval and document grouping. High-dimensional trained dense vectors have received much attention in recent years as such representations. We investigate the structure of semantic spaces that arise from embeddings made with Sentence-BERT and find that the representations suffer from a well-known problem in high dimensions called hubness. Hubness results in asymmetric neighborhood relations, such that some texts (the hubs) are neighbours of many other texts while most texts (so-called anti-hubs), are neighbours of few or no other texts. We quantify the semantic quality of the embeddings using hubness scores and error rate of a neighbourhood based classifier. We find that when hubness is high, we can reduce error rate and hubness using hubness reduction methods. We identify a combination of two methods as resulting in the best reduction. For example, on one of the tested pretrained models, this combined method can reduce hubness by about 75% and error rate by about 9%. Thus, we argue that mitigating hubness in the embedding space provides better semantic representations of text.