Bridging Model-based Safety and Model-free Reinforcement Learning through System Identification of Low Dimensional Linear Models

TL;DR

This paper presents a method to combine model-based safety guarantees with model-free reinforcement learning by identifying low-dimensional linear models of complex robotic systems, demonstrated on a bipedal robot.

Contribution

It introduces a novel approach to extract linear low-dimensional models from high-dimensional nonlinear systems controlled by RL, enabling safety guarantees.

Findings

Low-dimensional models accurately capture closed-loop dynamics.

Linear models are stable and decoupled across control inputs.

Safety guarantees can be applied using model predictive control with barrier functions.

Abstract

Bridging model-based safety and model-free reinforcement learning (RL) for dynamic robots is appealing since model-based methods are able to provide formal safety guarantees, while RL-based methods are able to exploit the robot agility by learning from the full-order system dynamics. However, current approaches to tackle this problem are mostly restricted to simple systems. In this paper, we propose a new method to combine model-based safety with model-free reinforcement learning by explicitly finding a low-dimensional model of the system controlled by a RL policy and applying stability and safety guarantees on that simple model. We use a complex bipedal robot Cassie, which is a high dimensional nonlinear system with hybrid dynamics and underactuation, and its RL-based walking controller as an example. We show that a low-dimensional dynamical model is sufficient to capture the dynamics of the closed-loop system. We demonstrate that this model is linear, asymptotically stable, and is decoupled across control input in all dimensions. We further exemplify that such linearity exists even when using different RL control policies. Such results point out an interesting direction to understand the relationship between RL and optimal control: whether RL tends to linearize the nonlinear system during training in some cases. Furthermore, we illustrate that the found linear model is able to provide guarantees by safety-critical optimal control framework, e.g., Model Predictive Control with Control Barrier Functions, on an example of autonomous navigation using Cassie while taking advantage of the agility provided by the RL-based controller.