Differentiable Robust LQR Layers

TL;DR

This paper introduces a differentiable robust LQR layer that enhances reinforcement and imitation learning by explicitly modeling uncertainty and stochasticity, enabling robust policy optimization in uncertain environments.

Contribution

It presents a novel differentiable layer for robust LQR optimization by reformulating it as a convex semi-definite program, facilitating end-to-end learning under uncertainty.

Findings

Improves policy robustness in uncertain environments

Achieves better performance than existing methods without explicit uncertainty modeling

Demonstrates effectiveness in imitation learning and dynamic programming tasks

Abstract

This paper proposes a differentiable robust LQR layer for reinforcement learning and imitation learning under model uncertainty and stochastic dynamics. The robust LQR layer can exploit the advantages of robust optimal control and model-free learning. It provides a new type of inductive bias for stochasticity and uncertainty modeling in control systems. In particular, we propose an efficient way to differentiate through a robust LQR optimization program by rewriting it as a convex program (i.e. semi-definite program) of the worst-case cost. Based on recent work on using convex optimization inside neural network layers, we develop a fully differentiable layer for optimizing this worst-case cost, i.e. we compute the derivative of a performance measure w.r.t the model's unknown parameters, model uncertainty and stochasticity parameters. We demonstrate the proposed method on imitation learning and approximate dynamic programming on stochastic and uncertain domains. The experiment results show that the proposed method can optimize robust policies under uncertain situations, and are able to achieve a significantly better performance than existing methods that do not model uncertainty directly.