Provably Safe Model-Based Meta Reinforcement Learning: An Abstraction-Based Approach

TL;DR

This paper introduces a novel approach for training provably safe neural network controllers for uncertain nonlinear systems that can generalize to new tasks, using finite-state abstractions to ensure safety guarantees.

Contribution

It presents an abstraction-based method for training and composing neural network controllers with provable safety guarantees in a meta reinforcement learning setting.

Findings

Successfully controlled a wheeled robot in unseen cluttered environments.

Provided theoretical safety guarantees for the composed neural network controllers.

Demonstrated generalization to new tasks not seen during training.

Abstract

While conventional reinforcement learning focuses on designing agents that can perform one task, meta-learning aims, instead, to solve the problem of designing agents that can generalize to different tasks (e.g., environments, obstacles, and goals) that were not considered during the design or the training of these agents. In this spirit, in this paper, we consider the problem of training a provably safe Neural Network (NN) controller for uncertain nonlinear dynamical systems that can generalize to new tasks that were not present in the training data while preserving strong safety guarantees. Our approach is to learn a set of NN controllers during the training phase. When the task becomes available at runtime, our framework will carefully select a subset of these NN controllers and compose them to form the final NN controller. Critical to our approach is the ability to compute a finite-state abstraction of the nonlinear dynamical system. This abstract model captures the behavior of the closed-loop system under all possible NN weights, and is used to train the NNs and compose them when the task becomes available. We provide theoretical guarantees that govern the correctness of the resulting NN. We evaluated our approach on the problem of controlling a wheeled robot in cluttered environments that were not present in the training data.