Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage

TL;DR

This paper introduces value-based offline RL algorithms with PAC guarantees under minimal coverage assumptions, leveraging novel minimax learning methods for accurate Q-function estimation.

Contribution

It proposes new offline RL algorithms that require only partial coverage and realizability, with convergence guarantees based on minimax optimization techniques.

Findings

Algorithms achieve PAC guarantees under single-policy coverage.

Novel minimax learning algorithms provide $L^2$-convergence guarantees.

Refined conditions for offline RL with entropy-regularized Q-functions.

Abstract

In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, Bellman completeness, and/or hard margin (gap). In this work we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms to accurately estimate soft or vanilla Q-functions with $L^2$-convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.