Synthesizing Neural Network Controllers with Probabilistic Model based Reinforcement Learning

TL;DR

This paper introduces a data-efficient, model-based reinforcement learning algorithm that uses neural network dynamics with calibrated uncertainty for rapid controller learning in robotics, outperforming existing methods in complex tasks.

Contribution

The paper proposes a novel neural network dynamics model with variational dropout and techniques to improve convergence, enhancing data efficiency and scalability in controller synthesis.

Findings

Competitive data-efficiency with PILCO

Successful learning of complex neural network controllers

Effective control of a six-legged underwater robot

Abstract

We present an algorithm for rapidly learning controllers for robotics systems. The algorithm follows the model-based reinforcement learning paradigm, and improves upon existing algorithms; namely Probabilistic learning in Control (PILCO) and a sample-based version of PILCO with neural network dynamics (Deep-PILCO). We propose training a neural network dynamics model using variational dropout with truncated Log-Normal noise. This allows us to obtain a dynamics model with calibrated uncertainty, which can be used to simulate controller executions via rollouts. We also describe set of techniques, inspired by viewing PILCO as a recurrent neural network model, that are crucial to improve the convergence of the method. We test our method on a variety of benchmark tasks, demonstrating data-efficiency that is competitive with PILCO, while being able to optimize complex neural network controllers. Finally, we assess the performance of the algorithm for learning motor controllers for a six legged autonomous underwater vehicle. This demonstrates the potential of the algorithm for scaling up the dimensionality and dataset sizes, in more complex control tasks.