Validating Wordscores

TL;DR

This paper rigorously evaluates the Wordscores method for estimating political party positions from texts, finding it generally fails to produce valid results across multiple validity tests.

Contribution

It provides a comprehensive validation of Wordscores, revealing its limitations and proposing conditions for improving its performance.

Findings

Wordscores does not reliably produce valid party position estimates.

Content validity shows issues with scored words in context.

Predictive validity for European Parliament groups is limited.

Abstract

Wordscores is a popular quantitative text scaling method to estimate parties' positions on a priori specified dimensions, without requiring the researchers to read or even understand the language in the documents they are analysing. This study tries to establish whereas Wordscores is able to deliver this promise by conducting a rigorous validation of its output using the Euromanifestos of 164 parties across 23 countries. We assess content validity by looking at the scored words in their context, criterion validity by comparing the Wordscores output to expert surveys and other judgemental estimates of party positions, and construct validity by using the Wordscores estimates to predict party membership in the European Parliament groups. We conclude that, despite the promises, Wordscores fails to deliver valid party positions, and outline three conditions under which its performance can be improved.