Empirical Study of Diachronic Word Embeddings for Scarce Data

TL;DR

This paper compares three models for learning diachronic word embeddings from scarce data, focusing on their ability to detect meaningful semantic drifts despite limited temporal information.

Contribution

It evaluates and contrasts incremental, dynamic filtering, and Bernoulli embedding models, highlighting their suitability and regularization strategies for scarce data scenarios.

Findings

Dynamic filtering performs best with limited data.

Regularization improves drift detection accuracy.

Model choice depends on data sparsity and drift characteristics.

Abstract

Word meaning change can be inferred from drifts of time-varying word embeddings. However, temporal data may be too sparse to build robust word embeddings and to discriminate significant drifts from noise. In this paper, we compare three models to learn diachronic word embeddings on scarce data: incremental updating of a Skip-Gram from Kim et al. (2014), dynamic filtering from Bamler and Mandt (2017), and dynamic Bernoulli embeddings from Rudolph and Blei (2018). In particular, we study the performance of different initialisation schemes and emphasise what characteristics of each model are more suitable to data scarcity, relying on the distribution of detected drifts. Finally, we regularise the loss of these models to better adapt to scarce data.