Deep Reinforcement Learning with Linear Quadratic Regulator Regions

TL;DR

This paper introduces a novel neural network-based control method that guarantees stability in nonlinear systems by combining linear quadratic regulator principles with neural network training, improving real-world transfer of simulated policies.

Contribution

The paper proposes a bias-shifted neural network approach that ensures stability regions in policy transfer from simulation to nonlinear real systems, bridging the gap between simulation and real-world control.

Findings

Successfully transferred policies from simulation to real inverted pendulum.

Guaranteed stability regions in nonlinear system control.

Enhanced robustness of policies in real-world environments.

Abstract

Practitioners often rely on compute-intensive domain randomization to ensure reinforcement learning policies trained in simulation can robustly transfer to the real world. Due to unmodeled nonlinearities in the real system, however, even such simulated policies can still fail to perform stably enough to acquire experience in real environments. In this paper we propose a novel method that guarantees a stable region of attraction for the output of a policy trained in simulation, even for highly nonlinear systems. Our core technique is to use "bias-shifted" neural networks for constructing the controller and training the network in the simulator. The modified neural networks not only capture the nonlinearities of the system but also provably preserve linearity in a certain region of the state space and thus can be tuned to resemble a linear quadratic regulator that is known to be stable for the real system. We have tested our new method by transferring simulated policies for a swing-up inverted pendulum to real systems and demonstrated its efficacy.