Unregularized limit of stochastic gradient method for Wasserstein distributionally robust optimization

TL;DR

This paper analyzes the unregularized limit of stochastic gradient methods in Wasserstein distributionally robust optimization, providing convergence guarantees and insights into the behavior of regularized approximations.

Contribution

It establishes the convergence of approximate gradients and critical points in Wasserstein DRO as regularization diminishes, extending analysis to general machine learning settings.

Findings

Convergence of approximate gradients over compact sets.

Critical points of regularized problems concentrate on original problem critical set.

Guarantees for projected stochastic gradient methods in unbounded sample spaces.

Abstract

Distributionally robust optimization offers a compelling framework for model fitting in machine learning, as it systematically accounts for data uncertainty. Focusing on Wasserstein distributionally robust optimization, we investigate the regularized problem where entropic smoothing yields a sampling-based approximation of the original objective. We establish the convergence of the approximate gradient over a compact set, leading to the concentration of the regularized problem critical points onto the original problem critical set as regularization diminishes and the number of approximation samples increases. Finally, we deduce convergence guarantees for a projected stochastic gradient method. Our analysis covers a general machine learning situation with an unbounded sample space and mixed continuous-discrete data.