A Statistical Model of Word Rank Evolution

TL;DR

This paper models word rank evolution over time using a Wright-Fisher inspired Markov Chain, revealing that actual linguistic changes are more complex than neutral models predict, with stable core words across languages.

Contribution

It introduces a mathematical framework for modeling word rank dynamics with a neutral evolutionary process and compares it to real data across multiple languages.

Findings

High-ranked words are more stable than low-ranked words.

Most words are rank stable but less so than the neutral model predicts.

Stopwords and Swadesh words show consistent rank stability across languages.

Abstract

The availability of large linguistic data sets enables data-driven approaches to study linguistic change. The Google Books corpus unigram frequency data set is used to investigate the word rank dynamics in eight languages. We observed the rank changes of the unigrams from 1900 to 2008 and compared it to a Wright-Fisher inspired model that we developed for our analysis. The model simulates a neutral evolutionary process with the restriction of having no disappearing and added words. This work explains the mathematical framework of the model - written as a Markov Chain with multinomial transition probabilities - to show how frequencies of words change in time. From our observations in the data and our model, word rank stability shows two types of characteristics: (1) the increase/decrease in ranks are monotonic, or (2) the rank stays the same. Based on our model, high-ranked words tend to be more stable while low-ranked words tend to be more volatile. Some words change in ranks in two ways: (a) by an accumulation of small increasing/decreasing rank changes in time and (b) by shocks of increase/decrease in ranks. Most words in all of the languages we have looked at are rank stable, but not as stable as a neutral model would predict. The stopwords and Swadesh words are observed to be rank stable across eight languages indicating linguistic conformity in established languages. These signatures suggest unigram frequencies in all languages have changed in a manner inconsistent with a purely neutral evolutionary process.