Model-Ensemble Trust-Region Policy Optimization

TL;DR

This paper introduces ME-TRPO, a model-ensemble approach for reinforcement learning that reduces sample complexity and stabilizes training by managing model uncertainty, outperforming traditional model-free methods.

Contribution

The paper proposes a novel ensemble-based method, ME-TRPO, that improves stability and efficiency in deep model-based reinforcement learning.

Findings

Significantly reduces sample complexity on continuous control tasks.

Ensemble models effectively manage uncertainty and stabilize training.

Likelihood ratio derivatives outperform backpropagation through time in stability.

Abstract

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.