Bias Reduction in Sample-Based Optimization

TL;DR

This paper introduces smooth estimators for stochastic optimization that reduce bias and variance compared to traditional sample average methods, with theoretical guarantees and practical benefits demonstrated in various statistical problems.

Contribution

It proposes a novel smoothing approach for sample-based optimization, establishing consistency and bias reduction, and demonstrates its effectiveness through theoretical analysis and numerical experiments.

Findings

Smoothing reduces bias in optimal value estimation.

The new estimators often have smaller variance and mean-square error.

Conditions for bias reduction are satisfied in common statistical models.

Abstract

We consider stochastic optimization problems which use observed data to estimate essential characteristics of the random quantities involved. Sample average approximation (SAA) or empirical (plug-in) estimation are very popular ways to use data in optimization. It is well known that sample average optimization suffers from downward bias. We propose to use smooth estimators rather than empirical ones in optimization problems. We establish consistency results for the optimal value and the set of optimal solutions of the new problem formulation. The performance of the proposed approach is compared to SAA theoretically and numerically. We analyze the bias of the new problems and identify sufficient conditions for ensuring less biased estimation of the optimal value of the true problem. At the same time, the error of the new estimator remains controlled. We show that those conditions are satisfied for many popular statistical problems such as regression models, classification problems, and optimization problems with Average (Conditional) Value-at-Risk. We have observed that smoothing the least-squares objective in a regression problem by a normal kernel leads to a ridge regression. Our numerical experience shows that the new estimators frequently exhibit also smaller variance and smaller mean-square error than those of SAA.