Validating Wordscores
Bastiaan Bruinsma, Kostas Gemenis

TL;DR
This paper rigorously evaluates the Wordscores method for estimating political party positions from texts, finding it generally fails to produce valid results across multiple validity tests.
Contribution
It provides a comprehensive validation of Wordscores, revealing its limitations and proposing conditions for improving its performance.
Findings
Wordscores does not reliably produce valid party position estimates.
Content validity shows issues with scored words in context.
Predictive validity for European Parliament groups is limited.
Abstract
Wordscores is a popular quantitative text scaling method to estimate parties' positions on a priori specified dimensions, without requiring the researchers to read or even understand the language in the documents they are analysing. This study tries to establish whereas Wordscores is able to deliver this promise by conducting a rigorous validation of its output using the Euromanifestos of 164 parties across 23 countries. We assess content validity by looking at the scored words in their context, criterion validity by comparing the Wordscores output to expert surveys and other judgemental estimates of party positions, and construct validity by using the Wordscores estimates to predict party membership in the European Parliament groups. We conclude that, despite the promises, Wordscores fails to deliver valid party positions, and outline three conditions under which its performance can be…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsElectoral Systems and Political Participation · Political Influence and Corporate Strategies · Social Media and Politics
