Benchmarking Batch Deep Reinforcement Learning Algorithms

TL;DR

This paper benchmarks recent batch reinforcement learning algorithms on Atari, revealing many underperform compared to DQN and behavioral policies, and introduces a strong baseline with adapted Batch-Constrained Q-learning.

Contribution

It provides a unified benchmarking framework for batch RL algorithms on Atari and introduces an adapted Batch-Constrained Q-learning baseline.

Findings

Many algorithms underperform DQN and behavioral policy in batch setting.

The adapted Batch-Constrained Q-learning outperforms existing algorithms.

Benchmarking reveals limitations of current batch RL methods.

Abstract

Widely-used deep reinforcement learning algorithms have been shown to fail in the batch setting--learning from a fixed data set without interaction with the environment. Following this result, there have been several papers showing reasonable performances under a variety of environments and batch settings. In this paper, we benchmark the performance of recent off-policy and batch reinforcement learning algorithms under unified settings on the Atari domain, with data generated by a single partially-trained behavioral policy. We find that under these conditions, many of these algorithms underperform DQN trained online with the same amount of data, as well as the partially-trained behavioral policy. To introduce a strong baseline, we adapt the Batch-Constrained Q-learning algorithm to a discrete-action setting, and show it outperforms all existing algorithms at this task.