Should Models Be Accurate?

TL;DR

This paper questions the necessity of accurate environment models in model-based reinforcement learning, proposing a meta-learning approach that emphasizes usefulness over accuracy, leading to faster learning in non-stationary environments.

Contribution

It introduces a meta-learning algorithm that trains models based on their usefulness for learning, rather than their accuracy in simulating environment dynamics.

Findings

Faster learning in non-stationary environments with the proposed method

Accurate models are not always necessary or most useful

Meta-learning improves model utility for planning

Abstract

Model-based Reinforcement Learning (MBRL) holds promise for data-efficiency by planning with model-generated experience in addition to learning with experience from the environment. However, in complex or changing environments, models in MBRL will inevitably be imperfect, and their detrimental effects on learning can be difficult to mitigate. In this work, we question whether the objective of these models should be the accurate simulation of environment dynamics at all. We focus our investigations on Dyna-style planning in a prediction setting. First, we highlight and support three motivating points: a perfectly accurate model of environment dynamics is not practically achievable, is not necessary, and is not always the most useful anyways. Second, we introduce a meta-learning algorithm for training models with a focus on their usefulness to the learner instead of their accuracy in modelling the environment. Our experiments show that in a simple non-stationary environment, our algorithm enables faster learning than even using an accurate model built with domain-specific knowledge of the non-stationarity.