Critic Regularized Regression

TL;DR

This paper introduces Critic Regularized Regression (CRR), a novel offline reinforcement learning algorithm that effectively learns policies from fixed datasets, outperforming existing methods on complex benchmark tasks.

Contribution

The paper proposes CRR, a new critic-regularized regression approach for offline RL, demonstrating superior performance and scalability over state-of-the-art algorithms.

Findings

CRR outperforms existing offline RL algorithms on benchmark tasks.

CRR scales effectively to high-dimensional state and action spaces.

CRR demonstrates strong performance without online environment interaction.

Abstract

Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces -- outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.