Variational Inference MPC for Bayesian Model-based Reinforcement Learning

TL;DR

This paper introduces a Bayesian variational inference approach to model-based reinforcement learning, enhancing uncertainty modeling and improving performance in robotics tasks.

Contribution

It presents a novel variational inference MPC framework and a new probabilistic action ensemble method, PaETS, for better uncertainty handling in MBRL.

Findings

Improved asymptotic performance over PETS in locomotion tasks

Handles multimodal uncertainties in dynamics and trajectories

Reformulates stochastic methods like CEM in a Bayesian framework

Abstract

In recent studies on model-based reinforcement learning (MBRL), incorporating uncertainty in forward dynamics is a state-of-the-art strategy to enhance learning performance, making MBRLs competitive to cutting-edge model free methods, especially in simulated robotics tasks. Probabilistic ensembles with trajectory sampling (PETS) is a leading type of MBRL, which employs Bayesian inference to dynamics modeling and model predictive control (MPC) with stochastic optimization via the cross entropy method (CEM). In this paper, we propose a novel extension to the uncertainty-aware MBRL. Our main contributions are twofold: Firstly, we introduce a variational inference MPC, which reformulates various stochastic methods, including CEM, in a Bayesian fashion. Secondly, we propose a novel instance of the framework, called probabilistic action ensembles with trajectory sampling (PaETS). As a result, our Bayesian MBRL can involve multimodal uncertainties both in dynamics and optimal trajectories. In comparison to PETS, our method consistently improves asymptotic performance on several challenging locomotion tasks.