Apples to Oranges: Causal Effects of Answer Changing in Multiple-Choice Exams

TL;DR

This paper clarifies the causal effects of answer changing in multiple-choice exams using a formal framework, resolving debates by distinguishing effects on those who change answers versus those who do not.

Contribution

It introduces a potential outcomes framework to analyze answer changing effects, highlighting differences between treated and untreated examinees and clarifying previous controversies.

Findings

Answer changing has positive effects on those who change answers (ATT)

Answer changing has negative effects on those who do not change answers (ATU)

Answer changing and reviewing are distinct treatments with different effects

Abstract

Whether examinees' answer changing behavior while taking multiple-choice exams is beneficial or harmful is a long-standing puzzle in the educational and psychological measurement literature. Formalizing the problem using the potential outcomes framework, this article shows that the traditional method of comparing the proportions of "wrong to right" and "right to wrong" answer changing patterns--a method that has recently been criticized by van der Linden, Jeon, and Ferrara (2011)--indeed correctly identify the sign of the average answer changing effect, but only for those examinees who actually changed their initial responses. This subgroup effect is referred to as the average treatment effect on the treated (ATT) and generally differs from the average treatment effect on the untreated (ATU), that is, those who did not change their initial responses. Analyzing two real data sets, including van der Linden et al.'s (2011) controversial data, this article finds that the ATT of answer changing is positive while the ATU of answer changing is negative, therefore, the debate on answer changing effects can be easily resolved. The article also shows that answer changing and answer reviewing are two distinct treatments and knowing answer changing effects is not informative for predicting answer reviewing effects.