Stable Offline Value Function Learning with Bisimulation-based Representations

TL;DR

This paper introduces a bisimulation-based algorithm called KROPE that stabilizes offline value function learning by shaping state-action representations, leading to more accurate and stable policy evaluation in reinforcement learning.

Contribution

The paper proposes KROPE, a novel bisimulation-based method that stabilizes offline value function learning and improves accuracy over existing baselines.

Findings

KROPE learns stable state-action representations.

KROPE achieves lower value error than baseline methods.

Theoretical analysis confirms stability benefits of bisimulation-based approaches.

Abstract

In reinforcement learning, offline value function learning is the procedure of using an offline dataset to estimate the expected discounted return from each state when taking actions according to a fixed target policy. The stability of this procedure, i.e., whether it converges to its fixed-point, critically depends on the representations of the state-action pairs. Poorly learned representations can make value function learning unstable, or even divergent. Therefore, it is critical to stabilize value function learning by explicitly shaping the state-action representations. Recently, the class of bisimulation-based algorithms have shown promise in shaping representations for control. However, it is still unclear if this class of methods can \emph{stabilize} value function learning. In this work, we investigate this question and answer it affirmatively. We introduce a bisimulation-based algorithm called kernel representations for offline policy evaluation (\textsc{krope}). \textsc{krope} uses a kernel to shape state-action representations such that state-action pairs that have similar immediate rewards and lead to similar next state-action pairs under the target policy also have similar representations. We show that \textsc{krope}: 1) learns stable representations and 2) leads to lower value error than baselines. Our analysis provides new theoretical insight into the stability properties of bisimulation-based methods and suggests that practitioners can use these methods to improve the stability and accuracy of offline evaluation of reinforcement learning agents.