Handling Attrition in Longitudinal Studies: The Case for Refreshment Samples

TL;DR

This paper advocates for using refreshment samples in longitudinal panel studies to diagnose and correct attrition bias, demonstrating Bayesian and multiple imputation methods with real data examples.

Contribution

It introduces models for three-wave panels with refreshment samples, including nonterminal attrition, and analyzes bias in multiple imputation variance estimation.

Findings

Refreshment samples improve bias detection and adjustment in panel studies.

Bayesian and multiple imputation methods are effective for analyzing combined data.

Identified bias in standard multiple imputation variance estimators.

Abstract

Panel studies typically suffer from attrition, which reduces sample size and can result in biased inferences. It is impossible to know whether or not the attrition causes bias from the observed panel data alone. Refreshment samples - new, randomly sampled respondents given the questionnaire at the same time as a subsequent wave of the panel - offer information that can be used to diagnose and adjust for bias due to attrition. We review and bolster the case for the use of refreshment samples in panel studies. We include examples of both a fully Bayesian approach for analyzing the concatenated panel and refreshment data, and a multiple imputation approach for analyzing only the original panel. For the latter, we document a positive bias in the usual multiple imputation variance estimator. We present models appropriate for three waves and two refreshment samples, including nonterminal attrition. We illustrate the three-wave analysis using the 2007-2008 Associated Press-Yahoo! News Election Poll.