Missing data and bias in physics education research: A case for using multiple imputation

TL;DR

This paper highlights the bias introduced by complete-case analysis in physics education research due to missing data and advocates for using multiple imputation to improve accuracy and reduce bias.

Contribution

The study demonstrates that multiple imputation outperforms complete-case analysis in reducing bias in PER studies with missing data, advocating for its adoption.

Findings

Complete-case analysis introduces significant bias.

Multiple imputation provides more accurate results.

Using MI can reveal true differences between student groups.

Abstract

Physics education researchers (PER) commonly use complete-case analysis to address missing data. For complete-case analysis, researchers discard all data from any student who is missing any data. Despite its frequent use, no PER article we reviewed that used complete-case analysis provided evidence that the data met the assumption of missing completely at random (MCAR) necessary to ensure accurate results. Not meeting this assumption raises the possibility that prior studies have reported biased results with inflated gains that may obscure differences across courses. To test this possibility, we compared the accuracy of complete-case analysis and multiple imputation (MI) using simulated data. We simulated the data based on prior studies such that students who earned higher grades participated at higher rates, which made the data missing at random (MAR). PER studies seldom use MI, but MI uses all available data, has less stringent assumptions, and is more accurate and more statistically powerful than complete-case analysis. Results indicated that complete-case analysis introduced more bias than MI and this bias was large enough to obscure differences between student populations or between courses. We recommend that the PER community adopt the use of MI for handling missing data to improve the accuracy in research studies.