Mildly Constrained Evaluation Policy for Offline Reinforcement Learning

TL;DR

This paper introduces a Mildly Constrained Evaluation Policy (MCEP) for offline RL that uses a less restrictive policy during test time, improving performance across various benchmarks and integrating seamlessly with existing algorithms.

Contribution

The paper proposes MCEP, a novel approach that adjusts constraints for test time inference in offline RL, enhancing performance and compatibility with prior methods.

Findings

MCEP significantly improves performance on D4RL benchmarks.

MCEP can be integrated with existing offline RL algorithms as a plug-in.

Empirical results show MCEP enhances state-of-the-art methods.

Abstract

Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a \textit{Mildly Constrained Evaluation Policy (MCEP)} for test time inference with a more constrained \textit{target policy} for value estimation. Since the \textit{target policy} has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3BC (Fujimoto & Gu, 2021), AWAC (Nair et al., 2020) and DQL (Wang et al., 2023) algorithms. The empirical results on D4RL MuJoCo locomotion, high-dimensional humanoid and a set of 16 robotic manipulation tasks show that the MCEP brought significant performance improvement on classic offline RL methods and can further improve SOTA methods. The codes are open-sourced at \url{https://github.com/egg-west/MCEP.git}.