Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings

TL;DR

This paper investigates the variance of contextualized word embeddings across different contexts, revealing factors influencing this variance and proposing methods to mitigate position bias in sense disambiguation.

Contribution

It quantifies sense-wise variance in embeddings, analyzes influencing factors, and introduces a simple approach to reduce position bias in word sense disambiguation.

Findings

Embeddings can be highly consistent across contexts.

Part-of-speech and sentence length influence variance.

Position bias affects similarity of first words in contexts.

Abstract

Contextualized word embeddings in language models have given much advance to NLP. Intuitively, sentential information is integrated into the representation of words, which can help model polysemy. However, context sensitivity also leads to the variance of representations, which may break the semantic consistency for synonyms. We quantify how much the contextualized embeddings of each word sense vary across contexts in typical pre-trained models. Results show that contextualized embeddings can be highly consistent across contexts. In addition, part-of-speech, number of word senses, and sentence length have an influence on the variance of sense representations. Interestingly, we find that word representations are position-biased, where the first words in different contexts tend to be more similar. We analyze such a phenomenon and also propose a simple way to alleviate such bias in distance-based word sense disambiguation settings.