Robustifying Reinforcement Learning Policies with $\mathcal{L}_1$ Adaptive Control

TL;DR

This paper introduces a method to enhance the robustness of pre-trained reinforcement learning policies against dynamic variations using $ ext{L}_1$ adaptive control, avoiding the conservativeness of traditional robust training.

Contribution

The paper proposes a novel approach combining $ ext{L}_1$ adaptive control with pre-trained RL policies to improve robustness without retraining in varied environments.

Findings

Significant robustness improvements demonstrated in simulations.

Effective compensation for dynamic variations in real-world scenarios.

Outperforms traditional robust training methods in certain benchmarks.

Abstract

A reinforcement learning (RL) policy trained in a nominal environment could fail in a new/perturbed environment due to the existence of dynamic variations. Existing robust methods try to obtain a fixed policy for all envisioned dynamic variation scenarios through robust or adversarial training. These methods could lead to conservative performance due to emphasis on the worst case, and often involve tedious modifications to the training environment. We propose an approach to robustifying a pre-trained non-robust RL policy with $\mathcal{L}_1$ adaptive control. Leveraging the capability of an $\mathcal{L}_1$ control law in the fast estimation of and active compensation for dynamic variations, our approach can significantly improve the robustness of an RL policy trained in a standard (i.e., non-robust) way, either in a simulator or in the real world. Numerical experiments are provided to validate the efficacy of the proposed approach.