Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning

TL;DR

This paper introduces a hybrid reinforcement learning approach combining neural network-based model predictive control with model-free fine-tuning, significantly improving sample efficiency and performance in robotic locomotion tasks.

Contribution

It demonstrates that neural network dynamics models can be effectively integrated with MPC for sample-efficient model-based RL and used to initialize model-free learning for better performance.

Findings

Model-based approach achieves excellent sample efficiency with simple random data.

Hybrid method accelerates model-free learning by 3-5x on various locomotion benchmarks.

Neural network dynamics models enable stable and plausible robotic gaits.

Abstract

Model-free deep reinforcement learning algorithms have been shown to be capable of learning a wide range of robotic skills, but typically require a very large number of samples to achieve good performance. Model-based algorithms, in principle, can provide for much more efficient learning, but have proven difficult to extend to expressive, high-capacity models such as deep neural networks. In this work, we demonstrate that medium-sized neural network models can in fact be combined with model predictive control (MPC) to achieve excellent sample complexity in a model-based reinforcement learning algorithm, producing stable and plausible gaits to accomplish various complex locomotion tasks. We also propose using deep neural network dynamics models to initialize a model-free learner, in order to combine the sample efficiency of model-based approaches with the high task-specific performance of model-free methods. We empirically demonstrate on MuJoCo locomotion tasks that our pure model-based approach trained on just random action data can follow arbitrary trajectories with excellent sample efficiency, and that our hybrid algorithm can accelerate model-free learning on high-speed benchmark tasks, achieving sample efficiency gains of 3-5x on swimmer, cheetah, hopper, and ant agents. Videos can be found at https://sites.google.com/view/mbmf