What do you mean, BERT? Assessing BERT as a Distributional Semantics Model

TL;DR

This paper evaluates BERT's ability to serve as a distributional semantics model, revealing that while it shows some semantic coherence, sentence context influences embeddings in unintended ways.

Contribution

It provides an empirical assessment of BERT's semantic properties, highlighting limitations in its representation of word similarity and contextual influence.

Findings

BERT exhibits some semantic coherence but is not fully aligned with distributional semantics expectations.

Sentence position influences word embeddings despite lacking semantic meaning.

Contextual effects in BERT can distort similarity relationships.

Abstract

Contextualized word embeddings, i.e. vector representations for words in context, are naturally seen as an extension of previous noncontextual distributional semantic models. In this work, we focus on BERT, a deep neural network that produces contextualized embeddings and has set the state-of-the-art in several semantic tasks, and study the semantic coherence of its embedding space. While showing a tendency towards coherence, BERT does not fully live up to the natural expectations for a semantic vector space. In particular, we find that the position of the sentence in which a word occurs, while having no meaning correlates, leaves a noticeable trace on the word embeddings and disturbs similarity relationships.