ELMo and BERT in semantic change detection for Russian

TL;DR

This paper evaluates the effectiveness of ELMo and BERT contextualized embeddings in detecting semantic changes in Russian nouns and adjectives across different historical periods, comparing various aggregation and learning methods.

Contribution

It introduces a comprehensive evaluation of ELMo and BERT for diachronic semantic change detection in Russian, including aggregation techniques and supervised versus unsupervised methods.

Findings

ELMo and BERT outperform traditional methods in ranking semantic change.

Supervised techniques show improved accuracy over unsupervised approaches.

Aggregation methods significantly impact detection performance.

Abstract

We study the effectiveness of contextualized embeddings for the task of diachronic semantic change detection for Russian language data. Evaluation test sets consist of Russian nouns and adjectives annotated based on their occurrences in texts created in pre-Soviet, Soviet and post-Soviet time periods. ELMo and BERT architectures are compared on the task of ranking Russian words according to the degree of their semantic change over time. We use several methods for aggregation of contextualized embeddings from these architectures and evaluate their performance. Finally, we compare unsupervised and supervised techniques in this task.