Stability-certified reinforcement learning: A control-theoretic perspective

TL;DR

This paper presents a control-theoretic approach to certify the stability of reinforcement learning policies when integrated with nonlinear dynamical systems, ensuring robust stability through input-output gradient regulation and semidefinite programming.

Contribution

It introduces a novel method to certify stability of RL policies using gradient regulation and semidefinite programming, with empirical validation on control tasks.

Findings

The method certifies a large set of stabilizing controllers.

Reinforcement learning agents perform well within the stability-certified region.

Agents exhibit stable long-term learning behaviors.

Abstract

We investigate the important problem of certifying stability of reinforcement learning policies when interconnected with nonlinear dynamical systems. We show that by regulating the input-output gradients of policies, strong guarantees of robust stability can be obtained based on a proposed semidefinite programming feasibility problem. The method is able to certify a large set of stabilizing controllers by exploiting problem-specific structures; furthermore, we analyze and establish its (non)conservatism. Empirical evaluations on two decentralized control tasks, namely multi-flight formation and power system frequency regulation, demonstrate that the reinforcement learning agents can have high performance within the stability-certified parameter space, and also exhibit stable learning behaviors in the long run.