Matching Bounds: How Choice of Matching Algorithm Impacts Treatment Effects Estimates and What to Do about It

TL;DR

This paper examines how different matching algorithms can lead to varying treatment effect estimates in social science research, introduces a method to assess the robustness of these estimates, and demonstrates its application through replication studies.

Contribution

It introduces Matching Bounds, a novel finite-sample method to evaluate the robustness of treatment effect estimates against different matching algorithms.

Findings

Matching algorithms can produce inconsistent treatment effect estimates.

Matching Bounds help determine if results are robust to matching choices.

In one replication, results were robust; in another, they were not.

Abstract

Many major works in social science employ matching to make causal conclusions, but different matches on the same data may produce different treatment effect estimates, even when they achieve similar balance or minimize the same loss function. We discuss reasons and consequences of this problem. We present evidence of this problem by replicating ten papers that use matching and we find that different popular matching algorithms produce inconsistent results. We introduce Matching Bounds: a finite-sample, nonstochastic method that allows analysts to know whether a matched sample that produces different results with the same levels of balance and overall match quality could be obtained from their data. We apply Matching Bounds to a replication of two studies and show that in one case results are robust to this issue and in another they are not.