Sample Average Approximation for Distributionally Robust Optimization with $\phi$-divergences

TL;DR

This paper analyzes the sample complexity of distributionally robust optimization using $\,phi$-divergences, showing how divergence growth affects the feasibility of $P$-independent estimation.

Contribution

It characterizes the conditions under which sample average approximation achieves $P$-independent complexity based on divergence growth rates.

Findings

Superlinear divergence growth yields $P$-independent sample complexity.

Lower bounds demonstrate the optimality of the derived bounds.

For non-superlinear divergences, $P$-dependent complexity can be arbitrarily large.

Abstract

It is well known that estimating the expectation of any given bounded random variable with values in $[-B, B]$ has a sample complexity of $\mathrm{O}(B^2/\epsilon^2)$ that is independent of the underlying probability measure. We show that this property can no longer hold when evaluating the worst-case expectation of the random variable, where the probability measures defining the expectation belong to a $\phi$-divergence ball centered at some nominal measure $P$. Specifically, the sample complexity and its dependence on the nominal measure can be completely characterized by the growth of the divergence function. When the divergence function $\phi$ exhibits superlinear growth, a $P$-independent sample complexity can be obtained for sample average approximation, which depends only on the growth of $\phi$, the radius of the divergence ball, and the target precision. We also provide sample complexity lower bounds and demonstrate the optimality of the obtained bounds for commonly used $\phi$-divergences. On the other hand, when superlinear growth does not hold for $\phi$, we show that for any estimation method, evaluating the worst-case expectation has a $P$-dependent sample complexity lower bound that can be made arbitrarily large by changing $P$.