A Primal-Dual Algorithm for Offline Constrained Reinforcement Learning with Linear MDPs

TL;DR

This paper introduces a computationally efficient primal-dual algorithm for offline reinforcement learning with linear MDPs, achieving optimal sample complexity under weaker data coverage assumptions and extending to constrained RL settings.

Contribution

The paper presents the first efficient algorithm for offline RL with linear MDPs that attains $O(rac{1}{\

Findings

Achieves $O(rac{1}{\

Achieves $O(rac{1}{\

paper_type":"method"}}}

Abstract

We study offline reinforcement learning (RL) with linear MDPs under the infinite-horizon discounted setting which aims to learn a policy that maximizes the expected discounted cumulative reward using a pre-collected dataset. Existing algorithms for this setting either require a uniform data coverage assumptions or are computationally inefficient for finding an $\epsilon$-optimal policy with $O(\epsilon^{-2})$ sample complexity. In this paper, we propose a primal dual algorithm for offline RL with linear MDPs in the infinite-horizon discounted setting. Our algorithm is the first computationally efficient algorithm in this setting that achieves sample complexity of $O(\epsilon^{-2})$ with partial data coverage assumption. Our work is an improvement upon a recent work that requires $O(\epsilon^{-4})$ samples. Moreover, we extend our algorithm to work in the offline constrained RL setting that enforces constraints on additional reward signals.