Safe Policy Improvement with Baseline Bootstrapping

TL;DR

This paper introduces SPIBB, a safe policy improvement method in batch reinforcement learning that guarantees baseline performance, utilizing a bootstrapping approach based on uncertainty, and demonstrates its effectiveness in various domains including deep RL.

Contribution

The paper proposes SPIBB, a novel safe policy improvement algorithm with theoretical guarantees, practical variants, and a model-free deep RL implementation that outperforms existing methods in safety and performance.

Findings

SPIBB guarantees baseline performance in batch RL.

SPIBB outperforms existing algorithms in safety and mean performance.

Deep RL version SPIBB-DQN trains efficiently without environment interaction.

Abstract

This paper considers Safe Policy Improvement (SPI) in Batch Reinforcement Learning (Batch RL): from a fixed dataset and without direct access to the true environment, train a policy that is guaranteed to perform at least as well as the baseline policy used to collect the data. Our approach, called SPI with Baseline Bootstrapping (SPIBB), is inspired by the knows-what-it-knows paradigm: it bootstraps the trained policy with the baseline when the uncertainty is high. Our first algorithm, $\Pi_b$-SPIBB, comes with SPI theoretical guarantees. We also implement a variant, $\Pi_{\leq b}$-SPIBB, that is even more efficient in practice. We apply our algorithms to a motivational stochastic gridworld domain and further demonstrate on randomly generated MDPs the superiority of SPIBB with respect to existing algorithms, not only in safety but also in mean performance. Finally, we implement a model-free version of SPIBB and show its benefits on a navigation task with deep RL implementation called SPIBB-DQN, which is, to the best of our knowledge, the first RL algorithm relying on a neural network representation able to train efficiently and reliably from batch data, without any interaction with the environment.