Comparative Evaluation of Difference in Differences Methods for Staggered Adoption Interventions

TL;DR

This paper systematically compares various difference-in-differences methods for evaluating staggered healthcare interventions, highlighting their performance limitations and providing practical guidance for researchers.

Contribution

It offers a comprehensive review, simulation analysis, and real-world application of DiD methods tailored for staggered adoption, addressing gaps in existing comparisons.

Findings

Classical DiD methods can be biased with few clusters.

Performance improves with increasing number of clusters.

Practical recommendations for method selection are provided.

Abstract

Staggered adoption is a common approach for implementing healthcare interventions, where different units adopt the program at different times. Difference-in-differences (DiD) methods are frequently used to evaluate the effects of such interventions. Nonetheless, recent research has shown that classical DiD approaches designed for a single treatment start date can produce biased estimates in staggered adoption settings, particularly due to treatment effect heterogeneity across adoption and calendar time. Several alternative methods have been developed to address these limitations. However, these methods have not been fully systematically compared, and their practical utility remains unclear. Motivated by a payment program implemented by a healthcare provider in Hawaii, we provide a comprehensive review of the staggered adoption setting and a selection of DiD methods suitable for this context. We begin with a theoretical overview of these methods, followed by a simulation study designed to resemble the characteristics of our application, where the intervention is implemented at the cluster level. Our results show that the current methods tend to under-perform when the number of clusters is small, but improve as the number of clusters increases. We then apply the methods to evaluate the real-world payment program intervention and offer practical recommendations for researchers implementing DiD methods for staggered adoption settings. Finally, we translate our findings into practical guidance for applied researchers choosing among DiD methods for staggered adoption settings.