Beating the Winner's Curse via Inference-Aware Policy Optimization

TL;DR

This paper introduces inference-aware policy optimization, a method that balances estimated policy improvements with the likelihood of passing significance tests, thereby reducing the winner's curse in policy learning.

Contribution

It proposes a novel optimization strategy that accounts for downstream evaluation significance, characterizes the Pareto frontier of tradeoffs, and enables better policy selection.

Findings

Effectively reduces winner's curse in policy optimization.

Balances estimated improvements with statistical significance.

Demonstrates improved policy evaluation in simulations.

Abstract

There has been a surge of recent interest in automatically learning policies to target treatment decisions based on rich individual covariates. In addition, practitioners want confidence that the learned policy has better performance than the incumbent policy according to downstream policy evaluation. However, due to the winner's curse -- an issue where the policy optimization procedure exploits prediction errors rather than finding actual improvements -- predicted performance improvements are often not substantiated by downstream policy evaluation. To address this challenge, we propose a novel strategy called inference-aware policy optimization, which modifies policy optimization to account for how the policy will be evaluated downstream. Specifically, it optimizes not only for the estimated objective value, but also for the chances that the estimate of the policy's improvement passes a significance test during downstream policy evaluation. We mathematically characterize the Pareto frontier of policies according to the tradeoff of these two goals. Based on our characterization, we design a policy optimization algorithm that estimates the Pareto frontier using machine learning models; then, the decision-maker can select the policy that optimizes their desired tradeoff, after which policy evaluation can be performed on the test set as usual. Finally, we perform simulations to illustrate the effectiveness of our methodology.