Optimizer's Information Criterion: Dissecting and Correcting Bias in Data-Driven Optimization

TL;DR

This paper introduces the Optimizer's Information Criterion (OIC), a novel bias correction method for data-driven optimization that accurately estimates true performance without additional computational cost, improving decision-making.

Contribution

The paper develops OIC, a generalization of AIC, to correct bias in data-driven optimization, applicable across various models and optimization contexts, without extra optimization solves.

Findings

OIC effectively reduces bias in empirical optimization results.

OIC outperforms traditional methods like cross-validation in accuracy.

Numerical experiments demonstrate OIC's superior performance on real-world datasets.

Abstract

In data-driven optimization, the sample performance of the obtained decision typically incurs an optimistic bias against the true performance, a phenomenon commonly known as the Optimizer's Curse and intimately related to overfitting in machine learning. Common techniques to correct this bias, such as cross-validation, require repeatedly solving additional optimization problems and are therefore computationally expensive. We develop a general bias correction approach, building on what we call Optimizer's Information Criterion (OIC), that directly approximates the first-order bias and does not require solving any additional optimization problems. Our OIC generalizes the celebrated Akaike Information Criterion to evaluate the objective performance in data-driven optimization, which crucially involves not only model fitting but also its interplay with the downstream optimization. As such it can be used for decision selection instead of only model selection. We apply our approach to a range of data-driven optimization formulations comprising empirical and parametric models, their regularized counterparts, and furthermore contextual optimization. Finally, we provide numerical validation on the superior performance of our approach under synthetic and real-world datasets.