On Globally Optimal Stochastic Policy Gradient Methods for Domain Randomized LQR Synthesis

TL;DR

This paper develops a theoretically grounded stochastic policy gradient method for domain randomized LQR synthesis, demonstrating convergence to global optima and improved controller robustness through repeated system sampling.

Contribution

It introduces a stochastic gradient descent approach for domain randomized LQR that guarantees convergence and enhances controller performance by leveraging repeated system sampling.

Findings

Gradients converge to global optima with proper hyperparameters.

Resampling systems at each iteration improves controller robustness.

Sampling is computationally efficient and yields lower variability.

Abstract

Domain randomization is a simple, effective, and flexible scheme for obtaining robust feedback policies aimed at reducing the sim-to-real gap due to model mismatch. While domain randomization methods have yielded impressive demonstrations in the robotics-learning literature, general and theoretically motivated principles for designing optimization schemes that effectively leverage the randomization are largely unexplored. We address this gap by considering a stochastic policy gradient descent method for the domain randomized linear-quadratic regulator synthesis problem, a situation simple enough to provide theoretical guarantees. In particular, we demonstrate that stochastic gradients obtained by repeatedly sampling new systems at each gradient step converge to global optima with appropriate hyperparameters choices, and yield better controllers with lower variability in the final controllers when compared to approaches that do not resample. Sampling is often a quick and cheap operation, so computing policy gradients with newly sampled systems at each iteration is preferable to evaluating gradients on a fixed set of systems.