Revisiting Bellman Errors for Offline Model Selection

TL;DR

This paper investigates the use of Bellman errors for offline model selection in reinforcement learning, explaining past limitations, identifying conditions for success, and proposing a new estimator that improves performance on various tasks.

Contribution

It clarifies why previous Bellman error methods underperformed, identifies conditions for their success, and introduces a more accurate MSBE estimator for better offline policy selection.

Findings

New MSBE estimator outperforms prior methods

Achieves strong results on Atari and other discrete control tasks

Provides theoretical insights into Bellman error-based model selection

Abstract

Offline model selection (OMS), that is, choosing the best policy from a set of many policies given only logged data, is crucial for applying offline RL in real-world settings. One idea that has been extensively explored is to select policies based on the mean squared Bellman error (MSBE) of the associated Q-functions. However, previous work has struggled to obtain adequate OMS performance with Bellman errors, leading many researchers to abandon the idea. To this end, we elucidate why previous work has seen pessimistic results with Bellman errors and identify conditions under which OMS algorithms based on Bellman errors will perform well. Moreover, we develop a new estimator of the MSBE that is more accurate than prior methods. Our estimator obtains impressive OMS performance on diverse discrete control tasks, including Atari games.