Assessing Inference Methods

TL;DR

This paper evaluates various simulation-based approaches for assessing the reliability of inference methods, highlighting their trade-offs, limitations, and proposing better alternatives for applied researchers.

Contribution

It provides a comprehensive analysis of inference assessment methods, identifies pitfalls in common practices, and offers practical recommendations and improved simulation strategies.

Findings

Different assessment methods have distinct strengths and weaknesses.

Common simulation practices can be misleading in shift-share designs.

The paper proposes improved simulation approaches for more reliable inference evaluation.

Abstract

We analyze different types of simulations that applied researchers can use to assess whether their inference methods reliably control false-positive rates. We show that different assessments involve trade-offs, varying in the types of problems they may detect, finite-sample performance, susceptibility to sequential-testing distortions, susceptibility to cherry-picking, and implementation complexity. We also show that a commonly used simulation to assess inference methods in shift-share designs can lead to misleading conclusions and propose alternatives. Overall, we provide novel insights and recommendations for applied researchers on how to choose, implement, and interpret inference assessments in their empirical applications.