Conditioning on the pre-test versus gain score modeling: revisiting the controversy in a multilevel setting

TL;DR

This paper compares two statistical approaches for estimating treatment effects in multilevel settings, analyzing their advantages and limitations through analytical results and simulations, especially considering individual and cluster-level treatments.

Contribution

It provides a detailed analysis of conditioning on pre-test versus gain score modeling in multilevel data, highlighting when each approach is preferable.

Findings

Conditioning approach is advantageous for large clusters with cluster-level treatment.

Gain score modeling performs well in individual-level treatment scenarios.

Cluster mean of pre-test score influences the effectiveness of the conditioning approach.

Abstract

We consider estimating the effect of a treatment on the progress of subjects tested both before and after treatment assignment. A vast literature compares the competing approaches of modeling the post-test score conditionally on the pre-test score versus modeling the difference, namely the gain score. Our contribution resides in analyzing the merits and drawbacks of the two approaches in a multilevel setting. This is relevant in many fields, for example education with students nested into schools. The multilevel structure raises peculiar issues related to the contextual effects and the distinction between individual-level and cluster-level treatment. We derive approximate analytical results and compare the two approaches by a simulation study. For an individual-level treatment our findings are in line with the literature, whereas for a cluster-level treatment we point out the key role of the cluster mean of the pre-test score, which favors the conditioning approach in settings with large clusters.