Word Embeddings: Stability and Semantic Change

TL;DR

This paper investigates the instability of popular word embedding techniques, introduces a statistical model and a new metric for measuring this instability, and proposes a method to reduce it for better tracking semantic change over time.

Contribution

It provides an experimental analysis of embedding instability, develops a statistical model and a novel instability metric, and offers a method to improve semantic change detection.

Findings

Embedding techniques show significant instability across runs.

A statistical model effectively describes embedding instability.

A modified averaging method reduces instability and improves semantic change detection.

Abstract

Word embeddings are computed by a class of techniques within natural language processing (NLP), that create continuous vector representations of words in a language from a large text corpus. The stochastic nature of the training process of most embedding techniques can lead to surprisingly strong instability, i.e. subsequently applying the same technique to the same data twice, can produce entirely different results. In this work, we present an experimental study on the instability of the training process of three of the most influential embedding techniques of the last decade: word2vec, GloVe and fastText. Based on the experimental results, we propose a statistical model to describe the instability of embedding techniques and introduce a novel metric to measure the instability of the representation of an individual word. Finally, we propose a method to minimize the instability - by computing a modified average over multiple runs - and apply it to a specific linguistic problem: The detection and quantification of semantic change, i.e. measuring changes in the meaning and usage of words over time.