Planning with Expectation Models for Control

TL;DR

This paper extends model-based reinforcement learning to control tasks using expectation models, proving the necessity of value function updates and exploring strategies for policy improvement with new algorithms.

Contribution

It introduces the first general algorithms for MBRL with expectation models in control, demonstrating the need for value updates and analyzing different planning strategies.

Findings

Planning with expectation models requires value-function updates.

Three strategies for policy improvement are proposed and analyzed.

Algorithms are validated through computational experiments.

Abstract

In model-based reinforcement learning (MBRL), Wan et al. (2019) showed conditions under which the environment model could produce the expectation of the next feature vector rather than the full distribution, or a sample thereof, with no loss in planning performance. Such expectation models are of interest when the environment is stochastic and non-stationary, and the model is approximate, such as when it is learned using function approximation. In these cases a full distribution model may be impractical and a sample model may be either more expensive computationally or of high variance. Wan et al. considered only planning for prediction to evaluate a fixed policy. In this paper, we treat the control case - planning to improve and find a good approximate policy. We prove that planning with an expectation model must update a state-value function, not an action-value function as previously suggested (e.g., Sorg & Singh, 2010). This opens the question of how planning influences action selections. We consider three strategies for this and present general MBRL algorithms for each. We identify the strengths and weaknesses of these algorithms in computational experiments. Our algorithms and experiments are the first to treat MBRL with expectation models in a general setting.