Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control

TL;DR

This paper introduces a data-efficient reinforcement learning framework using probabilistic model predictive control with Gaussian Processes, effectively handling constraints and reducing environment interactions.

Contribution

It presents a novel model-based RL approach that incorporates probabilistic models and theoretical guarantees for constrained environments, improving data efficiency.

Findings

Achieves state-of-the-art data efficiency in RL tasks.

Effectively handles state and control constraints.

Provides theoretical guarantees for optimality with Gaussian Process models.

Abstract

Trial-and-error based reinforcement learning (RL) has seen rapid advancements in recent times, especially with the advent of deep neural networks. However, the majority of autonomous RL algorithms require a large number of interactions with the environment. A large number of interactions may be impractical in many real-world applications, such as robotics, and many practical systems have to obey limitations in the form of state space or control constraints. To reduce the number of system interactions while simultaneously handling constraints, we propose a model-based RL framework based on probabilistic Model Predictive Control (MPC). In particular, we propose to learn a probabilistic transition model using Gaussian Processes (GPs) to incorporate model uncertainty into long-term predictions, thereby, reducing the impact of model errors. We then use MPC to find a control sequence that minimises the expected long-term cost. We provide theoretical guarantees for first-order optimality in the GP-based transition models with deterministic approximate inference for long-term planning. We demonstrate that our approach does not only achieve state-of-the-art data efficiency, but also is a principled way for RL in constrained environments.