Rotations and Interpretability of Word Embeddings: the Case of the Russian Language

TL;DR

This paper explores how orthogonal transformations can improve the interpretability and stability of word embedding components for Russian language models, enhancing their usefulness for linguistic analysis.

Contribution

It demonstrates that specific orthogonal transformations can enhance component interpretability and stability in Russian word embeddings, a novel approach in this context.

Findings

Orthogonal transformations can increase component interpretability.

Transformations improve stability of embeddings under re-learning.

Applicable to multiple Russian language embedding models.

Abstract

Consider a continuous word embedding model. Usually, the cosines between word vectors are used as a measure of similarity of words. These cosines do not change under orthogonal transformations of the embedding space. We demonstrate that, using some canonical orthogonal transformations from SVD, it is possible both to increase the meaning of some components and to make the components more stable under re-learning. We study the interpretability of components for publicly available models for the Russian language (RusVectores, fastText, RDT).