Bounding-Box Inference for Error-Aware Model-Based Reinforcement Learning

TL;DR

This paper introduces bounding-box inference, a method for estimating model uncertainty in model-based reinforcement learning, enabling more reliable and selective planning when models are inaccurate.

Contribution

It proposes bounding-box inference as a novel approach to quantify uncertainty, improving the effectiveness of selective planning in model-based RL.

Findings

Bounding-box inference reliably supports effective selective planning.

Distribution insensitive inference improves uncertainty estimation.

Selective planning reduces negative impact of model inaccuracies.

Abstract

In model-based reinforcement learning, simulated experiences from the learned model are often treated as equivalent to experience from the real environment. However, when the model is inaccurate, it can catastrophically interfere with policy learning. Alternatively, the agent might learn about the model's accuracy and selectively use it only when it can provide reliable predictions. We empirically explore model uncertainty measures for selective planning and show that best results require distribution insensitive inference to estimate the uncertainty over model-based updates. To that end, we propose and evaluate bounding-box inference, which operates on bounding-boxes around sets of possible states and other quantities. We find that bounding-box inference can reliably support effective selective planning.