Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change

TL;DR

This paper develops a methodology to quantify semantic change using word embeddings and uncovers two statistical laws of semantic evolution across multiple languages and centuries.

Contribution

It introduces a robust evaluation method for diachronic word embeddings and reveals two quantitative laws governing semantic change.

Findings

Semantic change rate scales with inverse power-law of word frequency

More polysemous words tend to change meanings faster

Methodology validated across six historical corpora in four languages

Abstract

Understanding how words change their meanings over time is key to models of language and cultural evolution, but historical data on meaning is scarce, making theories hard to develop and test. Word embeddings show promise as a diachronic tool, but have not been carefully evaluated. We develop a robust methodology for quantifying semantic change by evaluating word embeddings (PPMI, SVD, word2vec) against known historical changes. We then use this methodology to reveal statistical laws of semantic evolution. Using six historical corpora spanning four languages and two centuries, we propose two quantitative laws of semantic change: (i) the law of conformity---the rate of semantic change scales with an inverse power-law of word frequency; (ii) the law of innovation---independent of frequency, words that are more polysemous have higher rates of semantic change.