Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning

TL;DR

This paper introduces DPE, a novel offline reinforcement learning algorithm that combines sequence modeling with double policy estimation to reduce variance and improve performance on benchmark tasks.

Contribution

The paper proposes DPE, a new method integrating sequence modeling and double policy estimation with proven variance reduction, advancing offline RL performance.

Findings

DPE outperforms state-of-the-art baselines on several OpenAI Gym tasks.

The method demonstrates statistically proven variance reduction.

DPE effectively combines sequence modeling with double policy estimation for offline RL.

Abstract

Offline reinforcement learning aims to utilize datasets of previously gathered environment-action interaction records to learn a policy without access to the real environment. Recent work has shown that offline reinforcement learning can be formulated as a sequence modeling problem and solved via supervised learning with approaches such as decision transformer. While these sequence-based methods achieve competitive results over return-to-go methods, especially on tasks that require longer episodes or with scarce rewards, importance sampling is not considered to correct the policy bias when dealing with off-policy data, mainly due to the absence of behavior policy and the use of deterministic evaluation policies. To this end, we propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation (DPE) in a unified framework with statistically proven properties on variance reduction. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks. Our method brings a performance improvements on selected methods which outperforms SOTA baselines in several tasks, demonstrating the advantages of enabling double policy estimation for sequence-modeled reinforcement learning.