Invariance and identifiability issues for word embeddings

TL;DR

This paper explores how invariance and identifiability issues in word embeddings affect their comparability and performance evaluation, highlighting the impact of transformation classes that preserve different criteria functions.

Contribution

It formally analyzes the invariance and non-uniqueness of word embeddings and discusses how these issues influence performance assessments and comparisons.

Findings

Word embeddings are not unique due to invariance under certain transformations.

Disparities in task performance may stem from arbitrary choices within solution sets.

Formal treatment and numerical examples illustrate the impact of invariance issues.

Abstract

Word embeddings are commonly obtained as optimizers of a criterion function $f$ of a text corpus, but assessed on word-task performance using a different evaluation function $g$ of the test data. We contend that a possible source of disparity in performance on tasks is the incompatibility between classes of transformations that leave $f$ and $g$ invariant. In particular, word embeddings defined by $f$ are not unique; they are defined only up to a class of transformations to which $f$ is invariant, and this class is larger than the class to which $g$ is invariant. One implication of this is that the apparent superiority of one word embedding over another, as measured by word task performance, may largely be a consequence of the arbitrary elements selected from the respective solution sets. We provide a formal treatment of the above identifiability issue, present some numerical examples, and discuss possible resolutions.