Do Methodological Birds of a Feather Flock Together?

TL;DR

This paper formalizes and compares difference-in-differences and comparative interrupted time series methods within the potential outcomes framework, highlighting their similarities, differences, and practical implications for causal inference.

Contribution

It provides a formal mathematical comparison of CITS and DID methods, clarifies their equivalence under general conditions, and offers practical guidance for their application.

Findings

Most general CITS and DID estimate the same effects

Differences emerge under linearity or parallel trends assumptions

Recommendations for choosing flexible models and transparent assumptions

Abstract

Quasi-experimental methods have proliferated over the last two decades, as researchers develop causal inference tools for settings in which randomization is infeasible. Two popular such methods, difference-in-differences (DID) and comparative interrupted time series (CITS), compare observations before and after an intervention in a treated group to an untreated comparison group observed over the same period. Both methods rely on strong, untestable counterfactual assumptions. Despite their similarities, the methodological literature on CITS lacks the mathematical formality of DID. In this paper, we use the potential outcomes framework to formalize two versions of CITS - a general version described by Bloom (2005) and a linear version often used in health services research. We then compare these to two corresponding DID formulations - one with time fixed effects and one with time fixed effects and group trends. We also re-analyze three previously published studies using these methods. We demonstrate that the most general versions of CITS and DID impute the same counterfactuals and estimate the same treatment effects. The only difference between these two designs is the language used to describe them and their popularity in distinct disciplines. We also show that these designs diverge when one constrains them using linearity (CITS) or parallel trends (DID). We recommend defaulting to the more flexible versions and provide advice to practitioners on choosing between the more constrained versions by considering the data-generating mechanism. We also recommend greater attention to specifying the outcome model and counterfactuals in papers, allowing for transparent evaluation of the plausibility of causal assumptions.