Rotate King to get Queen: Word Relationships as Orthogonal Transformations in Embedding Space

TL;DR

This paper explores how orthogonal and linear transformations in embedding space can effectively model word relationships, offering an alternative to traditional vector arithmetic methods for analogy tasks.

Contribution

It introduces the use of orthogonal and linear transformations to represent word relationships, demonstrating their effectiveness compared to vector arithmetic.

Findings

Orthogonal transformations approximate word analogies with high accuracy.

Linear transformations outperform both orthogonal and vector arithmetic methods.

Transformations can serve as an alternative representation of word relationships.

Abstract

A notable property of word embeddings is that word relationships can exist as linear substructures in the embedding space. For example, $\textit{gender}$ corresponds to $\vec{\textit{woman}} - \vec{\textit{man}}$ and $\vec{\textit{queen}} - \vec{\textit{king}}$. This, in turn, allows word analogies to be solved arithmetically: $\vec{\textit{king}} - \vec{\textit{man}} + \vec{\textit{woman}} \approx \vec{\textit{queen}}$. This property is notable because it suggests that models trained on word embeddings can easily learn such relationships as geometric translations. However, there is no evidence that models $\textit{exclusively}$ represent relationships in this manner. We document an alternative way in which downstream models might learn these relationships: orthogonal and linear transformations. For example, given a translation vector for $\textit{gender}$, we can find an orthogonal matrix $R$, representing a rotation and reflection, such that $R(\vec{\textit{king}}) \approx \vec{\textit{queen}}$ and $R(\vec{\textit{man}}) \approx \vec{\textit{woman}}$. Analogical reasoning using orthogonal transformations is almost as accurate as using vector arithmetic; using linear transformations is more accurate than both. Our findings suggest that these transformations can be as good a representation of word relationships as translation vectors.