The Credibility Revolution in Political Science

TL;DR

This study analyzes how the credibility revolution has influenced political science research from 2003 to 2023, highlighting increased use of design-based methods and uneven adoption across journals and authors.

Contribution

It provides a large-scale classification of political science articles using language models, revealing trends in research design and credibility practices over two decades.

Findings

Rise of design-based studies with causal claims

Decline of model-based approaches relying on assumptions

Growth of survey experiments in credible research

Abstract

How has the credibility revolution shaped political science? We address this question by classifying 91,632 articles published between 2003 and 2023 across 156 political science journals using large language models, focusing on research design, credibility-enhancing practices, and citation patterns. We find that design-based studies -- those leveraging plausibly exogenous variation to justify causal claims -- have become increasingly common and receive a citation premium. In contrast, model-based approaches that rely on strong modeling assumptions have declined. Yet the rise of design-based work is uneven: it is concentrated in top journals and among authors at highly ranked institutions, and it is driven primarily by the growth of survey experiments. Other credibility-enhancing practices that help reduce false positives and false negatives, such as placebo tests and power calculations, remain rare. Taken together, our findings point to substantial but selective change, more consistent with a partial reform than a revolution.