A Case Study of Nonresponse Bias Analysis In Educational Assessment Surveys

TL;DR

This paper presents a comprehensive ten-step approach for analyzing nonresponse bias in educational surveys, applying it to a large longitudinal study and introducing novel indices to assess bias strength using pattern-mixture models.

Contribution

It develops a detailed framework for nonresponse bias analysis that accounts for both missing at random and not at random mechanisms, enhancing existing methods.

Findings

The approach effectively identifies and characterizes nonresponse bias in educational survey data.

Indices of bias provide quantifiable evidence about the impact of nonresponse.

The methodology is implementable with standard statistical software.

Abstract

Nonresponse bias is a widely prevalent problem for data on education. We develop a ten-step exemplar to guide nonresponse bias analysis (NRBA) in cross-sectional studies and apply these steps to the Early Childhood Longitudinal Study, Kindergarten Class of 2010-11. A key step is the construction of indices of nonresponse bias based on proxy pattern-mixture models for survey variables of interest. A novel feature is to characterize the strength of evidence about nonresponse bias contained in these indices, based on the strength of the relationship between the characteristics in the nonresponse adjustment and the key survey variables. Our NRBA improves existing methods by incorporating both missing at random and missing not at random mechanisms, and all analyses can be done straightforwardly with standard statistical software.