Efficient and Robust Reinforcement Learning with Uncertainty-based Value Expansion

TL;DR

This paper introduces RAVE, a hybrid reinforcement learning method that enhances robustness and performance by incorporating uncertainty modeling and risk aversion into value expansion techniques.

Contribution

The paper proposes RAVE, a novel hybrid RL algorithm that integrates probabilistic dynamics models and risk aversion to improve robustness and sample efficiency in stochastic environments.

Findings

RAVE outperforms previous methods in challenging environments.

Modeling uncertainty improves robustness of RL algorithms.

Achieved first place in NeurIPS 2019: Learn to Move.

Abstract

By integrating dynamics models into model-free reinforcement learning (RL) methods, model-based value expansion (MVE) algorithms have shown a significant advantage in sample efficiency as well as value estimation. However, these methods suffer from higher function approximation errors than model-free methods in stochastic environments due to a lack of modeling the environmental randomness. As a result, their performance lags behind the best model-free algorithms in some challenging scenarios. In this paper, we propose a novel Hybrid-RL method that builds on MVE, namely the Risk Averse Value Expansion (RAVE). With imaginative rollouts generated by an ensemble of probabilistic dynamics models, we further introduce the aversion of risks by seeking the lower confidence bound of the estimation. Experiments on a range of challenging environments show that by modeling the uncertainty completely, RAVE substantially enhances the robustness of previous model-based methods, and yields state-of-the-art performance. With this technique, our solution gets the first place in NeurIPS 2019: Learn to Move.