Clear Messages, Ambiguous Audiences: Measuring Interpretability in Political Communication

TL;DR

This study assesses interpretability in political communication, revealing high overall clarity but notable ambiguity in audience targeting, influenced by strategic incentives rather than random error.

Contribution

It introduces confidence-weighted human annotations for social media messages, uncovering structured measurement error linked to strategic political incentives.

Findings

Political messages are generally highly legible with confidence exceeding 0.99.

Audience classification confidence is reduced by 1.79 percentage points for constituency messages.

Interpretive uncertainty is systematically linked to strategic incentives rather than coder error.

Abstract

Text-based measurement in political research often treats classi6ication disagreement as random noise. We examine this assumption using con6idence-weighted human annotations of 5,000 social media messages by U.S. politicians. We 6ind that political communication is generally highly legible, with mean con6idence exceeding 0.99 across message type, partisan bias, and audience classi6ications. However, systematic variation concentrates in the constituency category, which exhibits a 1.79 percentage point penalty in audience classi6ication con6idence. Given the high baseline of agreement, this penalty represents a sharp relative increase in interpretive uncertainty. Within messages, intent remains clear while audience targeting becomes ambiguous. These patterns persist with politician 6ixed effects, suggesting that measurement error in political text is structured by strategic incentives rather than idiosyncratic coder error.