Winner's Curse Drives False Promises in Data-Driven Decisions: A Case Study in Refugee Matching

TL;DR

This paper highlights how the winner's curse leads to overly optimistic claims in data-driven policy evaluation, especially in refugee matching, by combining theoretical analysis and simulation to expose flaws in common methodologies.

Contribution

It demonstrates that common justifications do not prevent the winner's curse in model-based policy evaluation and provides empirical evidence from a refugee matching case study.

Findings

Model-based estimates can be highly optimistic despite justifications.

Simulations show large false benefits in refugee matching even when true effects are zero.

The winner's curse affects 55 papers, most relying on flawed evaluation methods.

Abstract

A major challenge in data-driven decision-making is accurate policy evaluation-i.e., guaranteeing that a learned decision-making policy achieves the promised benefits. A popular strategy is model-based policy evaluation, which estimates a model from data to infer counterfactual outcomes. This strategy is known to produce unwarrantedly optimistic estimates of the true benefit due to the winner's curse. We searched the recent literature on data-driven decision-making, identifying a sample of 55 papers published in the Management Science in the past decade; all but two relied on this flawed methodology. Several common justifications are provided: (1) the estimated models are accurate, stable, and well-calibrated, (2) the historical data uses random treatment assignment, (3) the model family is well-specified, and (4) the evaluation methodology uses sample splitting. Unfortunately, we show that no combination of these justifications avoids the winner's curse. First, we provide a theoretical analysis demonstrating that the winner's curse can cause large, spurious reported benefits even when all these justifications hold. Second, we perform a simulation study based on the recent and consequential data-driven refugee matching problem. We construct a synthetic refugee matching environment (calibrated to closely match the real setting) but designed so that no assignment policy can improve expected employment compared to random assignment. Model-based methods report large, stable gains of around 60% even when the true effect is zero; these gains are on par with improvements of 22-75% reported in the literature. Our results provide strong evidence against model-based evaluation.