# The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learning

## TL;DR

This paper introduces a theoretical framework and a novel algorithm, PDS, that leverage reward-free data to improve offline reinforcement learning performance while ensuring conservative and principled learning.

## Contribution

The paper presents the first theoretical analysis and a new algorithm, PDS, for using reward-free data in offline RL with provable guarantees.

## Key findings

- PDS significantly enhances offline RL performance with reward-free data.
- Theoretical analysis confirms the conservative nature of PDS.
- Empirical results demonstrate robustness across various tasks.

## Abstract

Self-supervised methods have become crucial for advancing deep learning by leveraging data itself to reduce the need for expensive annotations. However, the question of how to conduct self-supervised offline reinforcement learning (RL) in a principled way remains unclear. In this paper, we address this issue by investigating the theoretical benefits of utilizing reward-free data in linear Markov Decision Processes (MDPs) within a semi-supervised setting. Further, we propose a novel, Provable Data Sharing algorithm (PDS) to utilize such reward-free data for offline RL. PDS uses additional penalties on the reward function learned from labeled data to prevent overestimation, ensuring a conservative algorithm. Our results on various offline RL tasks demonstrate that PDS significantly improves the performance of offline RL algorithms with reward-free data. Overall, our work provides a promising approach to leveraging the benefits of unlabeled data in offline RL while maintaining theoretical guarantees. We believe our findings will contribute to developing more robust self-supervised RL methods.