Causal Inference in Counterbalanced Within-Subjects Designs

TL;DR

This paper examines the limitations of counterbalanced within-subjects experimental designs for causal inference, formalizes the assumptions needed, and offers practical guidance and alternative methods to improve validity.

Contribution

It introduces the concept of sequential exchangeability, formalizes the challenges of counterbalancing, and proposes diagnostic and design strategies for valid causal inference.

Findings

Counterbalancing assumptions are often unverifiable.

Sequential exchangeability is necessary for valid inference.

Alternative designs and diagnostics can improve causal validity.

Abstract

Experimental designs are fundamental for estimating causal effects. In some fields, within-subjects designs, which expose participants to both control and treatment at different time periods, are used to address practical and logistical concerns. Counterbalancing, a common technique in within-subjects designs, aims to remove carryover effects by randomizing treatment sequences. Despite its appeal, counterbalancing relies on the assumption that carryover effects are symmetric and cancel out, which is often unverifiable a priori. In this paper, we formalize the challenges of counterbalanced within-subjects designs using the potential outcomes framework. We introduce sequential exchangeability as an additional identification assumption necessary for valid causal inference in these designs. To address identification concerns, we propose diagnostic checks, the use of washout periods, and covariate adjustments, and alternative experimental designs to counterbalanced within-subjects design. Our findings demonstrate the limitations of counterbalancing and provide guidance on when and how within-subjects designs can be appropriately used for causal inference.