Another reason why normalized gain should continue to be used to analyze concept inventories (and estimate learning rates)

TL;DR

This paper defends the continued use of normalized gain for analyzing concept inventories, demonstrating that measurement errors cause biases and spurious correlations, and that proper estimation methods can yield unbiased learning rate assessments.

Contribution

The study clarifies the conditions under which normalized gain estimates are biased or unbiased, emphasizing the importance of measurement error considerations in learning rate analysis.

Findings

Measurement errors cause bias in the average ngain method.

The second estimation method remains unbiased in the presence of measurement errors.

Spurious pretest-ngain correlations are induced by measurement errors.

Abstract

A transformation called normalized gain (ngain) has been acknowledged as one of the most common measures of knowledge growth in pretest-posttest contexts in physics education research. Recent studies in math education have shown that ngains can also be applied to assess learners' ability to acquire unfamiliar knowledge, that is, to estimate their "learning rate". This quantity is estimated from learning data through two well-known methods: computing the average ngain of the group or computing the ngain of the average learner. These two methods commonly yield different results, and prior research has concluded that the difference between them is associated with a pretest-ngains correlation. Such a correlation would suggest a bias of this learning measurement because it implies its favoring of certain subgroups of students according to their performance in pretest measurements. The present study analyzes these two estimation methods by drawing on statistical models. Our results show that the two estimation methods are equivalent when no measurement errors exist. In contrast, when there are measurement errors, the first method provides a biased estimator, whereas the second one provides an unbiased estimator. Furthermore, these measurement errors induce a spurious correlation between the pretest and ngain scores. Our results seem consistent with prior research, except they show that measurement errors in pretest and posttest scores are the source of a spurious pretest-ngain correlation. Consequently, estimating learning rates might effectively provide unbiased estimates of knowledge change that control for the effect of prior knowledge even in the presence of pretest-ngain correlations.