Understanding and Addressing the Pitfalls of Bisimulation-based Representations in Offline Reinforcement Learning

TL;DR

This paper investigates why bisimulation-based state representations underperform in offline RL, identifies key issues like missing transitions and reward scaling, and proposes solutions that improve performance on benchmark tasks.

Contribution

It analyzes the pitfalls of bisimulation in offline RL, introduces an expectile operator and reward scaling strategies, and demonstrates improved results on benchmark datasets.

Findings

Bisimulation methods struggle with missing transitions in offline data.

Reward scaling is crucial to prevent feature collapse in representations.

Applying expectile operator and reward scaling improves performance on benchmarks.

Abstract

While bisimulation-based approaches hold promise for learning robust state representations for Reinforcement Learning (RL) tasks, their efficacy in offline RL tasks has not been up to par. In some instances, their performance has even significantly underperformed alternative methods. We aim to understand why bisimulation methods succeed in online settings, but falter in offline tasks. Our analysis reveals that missing transitions in the dataset are particularly harmful to the bisimulation principle, leading to ineffective estimation. We also shed light on the critical role of reward scaling in bounding the scale of bisimulation measurements and of the value error they induce. Based on these findings, we propose to apply the expectile operator for representation learning to our offline RL setting, which helps to prevent overfitting to incomplete data. Meanwhile, by introducing an appropriate reward scaling strategy, we avoid the risk of feature collapse in representation space. We implement these recommendations on two state-of-the-art bisimulation-based algorithms, MICo and SimSR, and demonstrate performance gains on two benchmark suites: D4RL and Visual D4RL. Codes are provided at \url{https://github.com/zanghyu/Offline_Bisimulation}.