Unbiased Gradient Estimation for Distributionally Robust Learning

TL;DR

This paper introduces a novel distributionally robust learning method that employs unbiased gradient estimation via multi-level Monte Carlo, improving model generalization with theoretical insights and practical benefits.

Contribution

It presents a new unbiased gradient estimation technique for DRL using multi-level Monte Carlo, balancing computational efficiency and variance reduction.

Findings

Significant improvement over previous DRL methods.

Theoretical analysis of gradient estimator optimality.

Empirical results demonstrating enhanced model robustness.

Abstract

Seeking to improve model generalization, we consider a new approach based on distributionally robust learning (DRL) that applies stochastic gradient descent to the outer minimization problem. Our algorithm efficiently estimates the gradient of the inner maximization problem through multi-level Monte Carlo randomization. Leveraging theoretical results that shed light on why standard gradient estimators fail, we establish the optimal parameterization of the gradient estimators of our approach that balances a fundamental tradeoff between computation time and statistical variance. Numerical experiments demonstrate that our DRL approach yields significant benefits over previous work.