Certifying Stability of Reinforcement Learning Policies using Generalized Lyapunov Functions

TL;DR

This paper introduces a method to certify the stability of reinforcement learning policies by learning generalized Lyapunov functions, relaxing classical conditions and enabling stability guarantees for complex systems.

Contribution

It proposes a novel approach to learn generalized Lyapunov functions from RL value functions, extending stability certification to nonlinear systems with learned policies.

Findings

Successfully certifies stability of RL policies on benchmark tasks

Extends Lyapunov methods with neural residuals for broader applicability

Achieves larger regions of attraction compared to classical Lyapunov approaches

Abstract

Establishing stability certificates for closed-loop systems under reinforcement learning (RL) policies is essential to move beyond empirical performance and offer guarantees of system behavior. Classical Lyapunov methods require a strict stepwise decrease in the Lyapunov function but such certificates are difficult to construct for learned policies. The RL value function is a natural candidate but it is not well understood how it can be adapted for this purpose. To gain intuition, we first study the linear quadratic regulator (LQR) problem and make two key observations. First, a Lyapunov function can be obtained from the value function of an LQR policy by augmenting it with a residual term related to the system dynamics and stage cost. Second, the classical Lyapunov decrease requirement can be relaxed to a generalized Lyapunov condition requiring only decrease on average over multiple time steps. Using this intuition, we consider the nonlinear setting and formulate an approach to learn generalized Lyapunov functions by augmenting RL value functions with neural network residual terms. Our approach successfully certifies the stability of RL policies trained on Gymnasium and DeepMind Control benchmarks. We also extend our method to jointly train neural controllers and stability certificates using a multi-step Lyapunov loss, resulting in larger certified inner approximations of the region of attraction compared to the classical Lyapunov approach. Overall, our formulation enables stability certification for a broad class of systems with learned policies by making certificates easier to construct, thereby bridging classical control theory and modern learning-based methods.