BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learning

TL;DR

BAIL is a new batch reinforcement learning algorithm that learns a value function to identify high-performing actions and trains a policy via imitation, outperforming existing methods in MuJoCo benchmarks.

Contribution

Introduces BAIL, a simple and effective batch DRL algorithm that combines value estimation with imitation learning for improved performance.

Findings

BAIL outperforms four other batch RL schemes on MuJoCo datasets.

BAIL is significantly faster computationally than batch Q-learning methods.

Experimental results demonstrate BAIL's superior policy quality.

Abstract

There has recently been a surge in research in batch Deep Reinforcement Learning (DRL), which aims for learning a high-performing policy from a given dataset without additional interactions with the environment. We propose a new algorithm, Best-Action Imitation Learning (BAIL), which strives for both simplicity and performance. BAIL learns a V function, uses the V function to select actions it believes to be high-performing, and then uses those actions to train a policy network using imitation learning. For the MuJoCo benchmark, we provide a comprehensive experimental study of BAIL, comparing its performance to four other batch Q-learning and imitation-learning schemes for a large variety of batch datasets. Our experiments show that BAIL's performance is much higher than the other schemes, and is also computationally much faster than the batch Q-learning schemes.