The 2008 election: A preregistered replication analysis

TL;DR

This paper conducts a series of preregistered replications of a 2013 analysis of 2008 US election polls, revealing potential issues with previous models and demonstrating the value of preregistration in historical and exploratory research.

Contribution

It introduces a rigorous preregistration approach to replicating complex election analysis, highlighting potential model limitations and emphasizing transparency in historical data analysis.

Findings

Potential model inadequately accounts for nonsampling error

Some patterns in earlier analysis may reflect noise

Demographic and geographic insights are validated

Abstract

We present an increasingly stringent set of replications of Ghitza & Gelman (2013), a multilevel regression and poststratification analysis of polls from the 2008 U.S. presidential election campaign, focusing on a set of plots showing the estimated Republican vote share for whites and for all voters, as a function of income level in each of the states. We start with a nearly-exact duplication that uses the posted code and changes only the model-fitting algorithm; we then replicate using already-analyzed data from 2004; and finally we set up preregistered replications using two surveys from 2008 that we had not previously looked at. We have already learned from our preliminary, non-preregistered replication, which has revealed a potential problem with the published analysis of Ghitza & Gelman (2013); it appears that our model may not sufficiently account for nonsampling error, and that some of the patterns presented in that earlier paper may simply reflect noise. In addition to the substantive interest in validating earlier findings about demographics, geography, and voting, the present project serves as a demonstration of preregistration in a setting where the subject matter is historical (and thus the replication data exist before the preregistration plan is written) and where the analysis is exploratory (and thus a replication cannot be simply deemed successful or unsuccessful based on the statistical significance of some particular comparison).