Substitution-based Semantic Change Detection using Contextual Embeddings

TL;DR

This paper introduces a simplified, interpretable, and efficient method for detecting semantic change using contextual embeddings and substitute words, outperforming previous approaches on standard datasets.

Contribution

The paper presents a novel substitution-based approach leveraging contextual embeddings for semantic change detection, improving efficiency, interpretability, and performance.

Findings

Outperforms existing methods on key datasets

More scalable and interpretable than prior approaches

Enables nuanced analysis of semantic change

Abstract

Measuring semantic change has thus far remained a task where methods using contextual embeddings have struggled to improve upon simpler techniques relying only on static word vectors. Moreover, many of the previously proposed approaches suffer from downsides related to scalability and ease of interpretation. We present a simplified approach to measuring semantic change using contextual embeddings, relying only on the most probable substitutes for masked terms. Not only is this approach directly interpretable, it is also far more efficient in terms of storage, achieves superior average performance across the most frequently cited datasets for this task, and allows for more nuanced investigation of change than is possible with static word vectors.