Post-Model-Selection Statistical Inference with Interrupted Time Series Designs: An Evaluation of an Assault Weapons Ban in California

TL;DR

This paper evaluates the use of interrupted time series analysis for causal inference in firearm sales data, highlighting methodological challenges and proposing improvements, but cautioning about the limitations of current approaches.

Contribution

It critically assesses standard statistical methods for interrupted time series analysis in firearm sales, proposing enhancements and emphasizing the limitations of causal inference in this context.

Findings

Standard methods face multiplicity and overfitting issues.

Proposed methodological improvements address some challenges.

Limitations remain in drawing definitive causal conclusions.

Abstract

There have been many claims in the media and a bit of respectable research about the causes of variation in firearm sales. The challenges for causal inference can be quite daunting. This paper reports an analysis of daily handgun sales in California from 1996 through 2018 using an interrupted time series design and analysis. The design was introduced to social scientists in 1963 by Campbell and Stanley, analysis methods were proposed by Box and Tiao in 1975, and more recent treatments are easily found (Box et al., 2016). But this approach to causal inference can be badly overmatched by the data on handgun sales, especially when the causal effects are estimated. More important for this paper are fundamental oversights in the standard statistical methods employed. Test multiplicity problems are introduced by adaptive model selection built into recommended practice. The challenges are computational and conceptual. Some progress is made on both problems that arguably improves on past research, but the take-home message may be to reduce aspirations about what can be learned.