Diverse Randomized Value Functions: A Provably Pessimistic Approach for Offline Reinforcement Learning

TL;DR

This paper introduces a novel offline RL method using diverse randomized value functions to provide robust, efficient, and provably pessimistic value estimates, improving policy learning from offline data.

Contribution

The paper proposes a diverse randomized value functions approach with a diversity regularization to improve uncertainty estimation and efficiency in offline RL, with theoretical guarantees.

Findings

Outperforms baseline methods in empirical tests

Reduces the number of networks needed for uncertainty estimation

Provides provable guarantees under linear MDP assumptions

Abstract

Offline Reinforcement Learning (RL) faces distributional shift and unreliable value estimation, especially for out-of-distribution (OOD) actions. To address this, existing uncertainty-based methods penalize the value function with uncertainty quantification and demand numerous ensemble networks, posing computational challenges and suboptimal outcomes. In this paper, we introduce a novel strategy employing diverse randomized value functions to estimate the posterior distribution of $Q$-values. It provides robust uncertainty quantification and estimates lower confidence bounds (LCB) of $Q$-values. By applying moderate value penalties for OOD actions, our method fosters a provably pessimistic approach. We also emphasize on diversity within randomized value functions and enhance efficiency by introducing a diversity regularization method, reducing the requisite number of networks. These modules lead to reliable value estimation and efficient policy learning from offline data. Theoretical analysis shows that our method recovers the provably efficient LCB-penalty under linear MDP assumptions. Extensive empirical results also demonstrate that our proposed method significantly outperforms baseline methods in terms of performance and parametric efficiency.