Achieving Semantic Consistency: Contextualized Word Representations for Political Text Analysis

TL;DR

This paper compares static Word2Vec and contextual BERT embeddings in political text analysis, demonstrating BERT's superior semantic stability over 20 years of news articles, suitable for tasks requiring consistent meaning interpretation.

Contribution

It provides an empirical comparison showing BERT's enhanced semantic stability over Word2Vec in political text analysis across a long time span.

Findings

BERT outperforms Word2Vec in semantic stability.

BERT captures subtle semantic variations.

BERT is more reliable for stable semantic analysis.

Abstract

Accurately interpreting words is vital in political science text analysis; some tasks require assuming semantic stability, while others aim to trace semantic shifts. Traditional static embeddings, like Word2Vec effectively capture long-term semantic changes but often lack stability in short-term contexts due to embedding fluctuations caused by unbalanced training data. BERT, which features transformer-based architecture and contextual embeddings, offers greater semantic consistency, making it suitable for analyses in which stability is crucial. This study compares Word2Vec and BERT using 20 years of People's Daily articles to evaluate their performance in semantic representations across different timeframes. The results indicate that BERT outperforms Word2Vec in maintaining semantic stability and still recognizes subtle semantic variations. These findings support BERT's use in text analysis tasks that require stability, where semantic changes are not assumed, offering a more reliable foundation than static alternatives.