Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words

TL;DR

This paper reveals that cosine similarity systematically underestimates the similarity of high-frequency words in BERT embeddings, due to differences in their representational geometry, impacting NLP tasks relying on such measures.

Contribution

It identifies and explains the systematic underestimation of high-frequency word similarities by cosine in BERT embeddings, linking it to frequency-related geometric differences.

Findings

Cosine similarity underestimates frequent word similarities compared to human judgments.

The underestimation is linked to differences in embedding geometry based on word frequency.

A formal geometric argument supports the observed effects.

Abstract

Cosine similarity of contextual embeddings is used in many NLP tasks (e.g., QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in which word similarities estimated by cosine over BERT embeddings are understated and trace this effect to training data frequency. We find that relative to human judgements, cosine similarity underestimates the similarity of frequent words with other instances of the same word or other words across contexts, even after controlling for polysemy and other factors. We conjecture that this underestimation of similarity for high frequency words is due to differences in the representational geometry of high and low frequency words and provide a formal argument for the two-dimensional case.