Learning the Dimensionality of Word Embeddings

TL;DR

This paper introduces a method for learning word embeddings with data-dependent, dynamic dimensionality, allowing the model to adapt the number of dimensions based on the data, providing insights into semantic distribution.

Contribution

It presents nonparametric variants of word2vec that learn the optimal embedding dimensionality, offering a new perspective on semantic representation in embeddings.

Findings

SD-SG and SD-CBOW are competitive with fixed-dimension models.

The models provide a distribution over embedding dimensionalities.

The approach offers insights into how semantics are distributed across dimensions.

Abstract

We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric analogs of Mikolov et al.'s (2013) well-known 'word2vec' models. Vector dimensionality is made dynamic by employing techniques used by Cote & Larochelle (2016) to define an RBM with an infinite number of hidden units. We show qualitatively and quantitatively that SD-SG and SD-CBOW are competitive with their fixed-dimension counterparts while providing a distribution over embedding dimensionalities, which offers a window into how semantics distribute across dimensions.