Safe Deep Model-Based Reinforcement Learning with Lyapunov Functions

TL;DR

This paper introduces a novel model-based reinforcement learning framework that incorporates Lyapunov functions to ensure safety and stability during training and policy execution, with theoretical guarantees and demonstrated effectiveness.

Contribution

It presents a new stability-augmented RL framework that learns Lyapunov functions and enforces safety constraints during policy learning, a significant advancement over prior methods.

Findings

Successfully enforces safety constraints during training

Provides mathematically provable stability guarantees

Demonstrates effectiveness through simulated experiments

Abstract

Model-based Reinforcement Learning (MBRL) has shown many desirable properties for intelligent control tasks. However, satisfying safety and stability constraints during training and rollout remains an open question. We propose a new Model-based RL framework to enable efficient policy learning with unknown dynamics based on learning model predictive control (LMPC) framework with mathematically provable guarantees of stability. We introduce and explore a novel method for adding safety constraints for model-based RL during training and policy learning. The new stability-augmented framework consists of a neural-network-based learner that learns to construct a Lyapunov function, and a model-based RL agent to consistently complete the tasks while satisfying user-specified constraints given only sub-optimal demonstrations and sparse-cost feedback. We demonstrate the capability of the proposed framework through simulated experiments.