The Least Restriction for Offline Reinforcement Learning

TL;DR

This paper introduces the Least Restriction framework for offline reinforcement learning, which minimizes constraints on action selection to improve stability and learning effectiveness from fixed datasets.

Contribution

The paper proposes a novel offline RL framework that reduces restrictions on action choices, addressing bootstrapping errors and enhancing learning from offline data.

Findings

LR can learn robustly from various offline datasets

LR outperforms previous methods on control tasks

LR avoids out-of-distribution actions effectively

Abstract

Many practical applications of reinforcement learning (RL) constrain the agent to learn from a fixed offline dataset of logged interactions, which has already been gathered, without offering further possibility for data collection. However, commonly used off-policy RL algorithms, such as the Deep Q Network and the Deep Deterministic Policy Gradient, are incapable of learning without data correlated to the distribution under the current policy, making them ineffective for this offline setting. As the first step towards useful offline RL algorithms, we analysis the reason of instability in standard off-policy RL algorithms. It is due to the bootstrapping error. The key to avoiding this error, is ensuring that the agent's action space does not go out of the fixed offline dataset. Based on our consideration, a creative offline RL framework, the Least Restriction (LR), is proposed in this paper. The LR regards selecting an action as taking a sample from the probability distribution. It merely set a little limit for action selection, which not only avoid the action being out of the offline dataset but also remove all the unreasonable restrictions in earlier approaches (e.g. Batch-Constrained Deep Q-Learning). In the further, we will demonstrate that the LR, is able to learn robustly from different offline datasets, including random and suboptimal demonstrations, on a range of practical control tasks.