Continuous Doubly Constrained Batch Reinforcement Learning

TL;DR

This paper introduces a novel batch reinforcement learning algorithm that effectively learns policies from fixed offline datasets by applying policy and value constraints, outperforming existing methods across multiple benchmarks.

Contribution

The paper proposes a new batch RL algorithm with policy and value constraints to address uncertainty and extrapolation issues in offline learning.

Findings

Outperforms state-of-the-art batch RL methods on 32 benchmarks

Effectively mitigates extrapolation errors in offline datasets

Works well regardless of data collection methods

Abstract

Reliant on too many experiments to learn good actions, current Reinforcement Learning (RL) algorithms have limited applicability in real-world settings, which can be too expensive to allow exploration. We propose an algorithm for batch RL, where effective policies are learned using only a fixed offline dataset instead of online interactions with the environment. The limited data in batch RL produces inherent uncertainty in value estimates of states/actions that were insufficiently represented in the training data. This leads to particularly severe extrapolation when our candidate policies diverge from one that generated the data. We propose to mitigate this issue via two straightforward penalties: a policy-constraint to reduce this divergence and a value-constraint that discourages overly optimistic estimates. Over a comprehensive set of 32 continuous-action batch RL benchmarks, our approach compares favorably to state-of-the-art methods, regardless of how the offline data were collected.