Analogies Explained: Towards Understanding Word Embeddings

TL;DR

This paper provides a probabilistically grounded explanation for the linear relationships observed in word embeddings, specifically addressing the analogy phenomenon in neural network-generated embeddings like word2vec.

Contribution

It introduces a formal probabilistic framework for understanding word transformations and proves the existence of linear relationships underlying analogies in word embeddings.

Findings

Linear relationships explain analogy phenomena in embeddings

Explicit error terms quantify deviations from ideal linearity

The framework offers a theoretical basis for embedding properties

Abstract

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to achieve it. Several explanations have been proposed, but each introduces assumptions that do not hold in practice. We derive a probabilistically grounded definition of paraphrasing that we re-interpret as word transformation, a mathematical description of "$w_x$ is to $w_y$". From these concepts we prove existence of linear relationships between W2V-type embeddings that underlie the analogical phenomenon, identifying explicit error terms.