Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity

TL;DR

This paper introduces Sparse-Reg, a sparsity-based regularization method that enhances offline reinforcement learning performance on small datasets by reducing overfitting, outperforming existing methods in continuous control tasks.

Contribution

The paper presents Sparse-Reg, a novel regularization technique leveraging sparsity to improve sample efficiency and mitigate overfitting in offline RL with limited data.

Findings

Sparse-Reg reduces overfitting on small datasets.

Sparse-Reg outperforms state-of-the-art baselines.

Effective in continuous control tasks.

Abstract

In this paper, we investigate the use of small datasets in the context of offline reinforcement learning (RL). While many common offline RL benchmarks employ datasets with over a million data points, many offline RL applications rely on considerably smaller datasets. We show that offline RL algorithms can overfit on small datasets, resulting in poor performance. To address this challenge, we introduce "Sparse-Reg": a regularization technique based on sparsity to mitigate overfitting in offline reinforcement learning, enabling effective learning in limited data settings and outperforming state-of-the-art baselines in continuous control.