On the Dimensionality of Word Embedding

TL;DR

This paper introduces a theoretical framework for understanding the optimal dimensionality of word embeddings, revealing a bias-variance trade-off and providing insights into robustness and overfitting.

Contribution

It proposes the PIP loss metric and applies matrix perturbation theory to explain and optimize the dimensionality of word embeddings.

Findings

Identifies a fundamental bias-variance trade-off in dimensionality selection

Explains the existence of an optimal embedding dimensionality

Provides conditions under which embeddings are robust to overfitting

Abstract

In this paper, we provide a theoretical understanding of word embedding and its dimensionality. Motivated by the unitary-invariance of word embedding, we propose the Pairwise Inner Product (PIP) loss, a novel metric on the dissimilarity between word embeddings. Using techniques from matrix perturbation theory, we reveal a fundamental bias-variance trade-off in dimensionality selection for word embeddings. This bias-variance trade-off sheds light on many empirical observations which were previously unexplained, for example the existence of an optimal dimensionality. Moreover, new insights and discoveries, like when and how word embeddings are robust to over-fitting, are revealed. By optimizing over the bias-variance trade-off of the PIP loss, we can explicitly answer the open question of dimensionality selection for word embedding.