How Robust are Robustness Checks?

TL;DR

This paper introduces a 'robustness radius' measure to quantify how much robustness checks differ from main estimates, enhancing transparency and interpretability in empirical research.

Contribution

It proposes a formal, interpretable measure for robustness checks using a bias framework and moment inequalities, addressing a gap in standard statistical procedures.

Findings

The robustness radius effectively quantifies differences in robustness checks.

It adapts to sampling uncertainty and correlation across regressions.

Application demonstrates its usefulness in guiding robustness judgments.

Abstract

Robustness checks are routine in empirical work, but there is no standard statistical procedure to formally measure what one can learn from them. I propose a "robustness radius" measure to quantify the amount by which the robustness checks estimands differ from the main specification estimand. I do so by framing robustness checks as explicitly biased regressions, clarifying what exactly the estimands are when comparing multiple regressions with slightly different samples, and applying a test from the moment inequalities literature. The robustness radius is easily interpretable and adapts to sampling uncertainty and correlation across regressions. An application shows that, although assessing overall robustness is context-specific, the robustness radius guides those judgments and improves transparency.