Dynamic Word Embeddings

TL;DR

This paper introduces a probabilistic dynamic word embedding model that captures how word meanings evolve over time, using latent trajectories and scalable inference algorithms, outperforming static models in interpretability and prediction.

Contribution

It presents a novel probabilistic framework with scalable inference for modeling temporal semantic evolution of words in text data.

Findings

More interpretable word trajectories

Higher predictive likelihoods than static models

Effective on multiple corpora

Abstract

We present a probabilistic language model for time-stamped text data which tracks the semantic evolution of individual words over time. The model represents words and contexts by latent trajectories in an embedding space. At each moment in time, the embedding vectors are inferred from a probabilistic version of word2vec [Mikolov et al., 2013]. These embedding vectors are connected in time through a latent diffusion process. We describe two scalable variational inference algorithms--skip-gram smoothing and skip-gram filtering--that allow us to train the model jointly over all times; thus learning on all data while simultaneously allowing word and context vectors to drift. Experimental results on three different corpora demonstrate that our dynamic model infers word embedding trajectories that are more interpretable and lead to higher predictive likelihoods than competing methods that are based on static models trained separately on time slices.