Model-Based Policy Search Using Monte Carlo Gradient Estimation with Real Systems Application

TL;DR

This paper introduces MC-PILCO, a model-based reinforcement learning algorithm using Gaussian Processes and Monte Carlo gradient estimation, demonstrating improved data efficiency and control in simulations and real systems.

Contribution

The paper presents MC-PILCO, a novel GP-based MBRL algorithm with structured kernels and Monte Carlo gradient estimation, optimized for real systems with partial observability.

Findings

MC-PILCO outperforms state-of-the-art GP-based MBRL algorithms in simulated environments.

MC-PILCO achieves effective control on real systems like Furuta pendulum and ball-and-plate.

Structured kernels and policy optimization techniques enhance data efficiency and control performance.

Abstract

In this paper, we present a Model-Based Reinforcement Learning (MBRL) algorithm named \emph{Monte Carlo Probabilistic Inference for Learning COntrol} (MC-PILCO). The algorithm relies on Gaussian Processes (GPs) to model the system dynamics and on a Monte Carlo approach to estimate the policy gradient. This defines a framework in which we ablate the choice of the following components: (i) the selection of the cost function, (ii) the optimization of policies using dropout, (iii) an improved data efficiency through the use of structured kernels in the GP models. The combination of the aforementioned aspects affects dramatically the performance of MC-PILCO. Numerical comparisons in a simulated cart-pole environment show that MC-PILCO exhibits better data efficiency and control performance w.r.t. state-of-the-art GP-based MBRL algorithms. Finally, we apply MC-PILCO to real systems, considering in particular systems with partially measurable states. We discuss the importance of modeling both the measurement system and the state estimators during policy optimization. The effectiveness of the proposed solutions has been tested in simulation and on two real systems, a Furuta pendulum and a ball-and-plate rig.