Reachability-Aware Laplacian Representation in Reinforcement Learning

TL;DR

This paper introduces Reachability-Aware Laplacian Representation (RA-LapRep), a scaled version of LapRep that better captures state reachability in reinforcement learning, improving reward shaping and bottleneck state discovery.

Contribution

The paper proposes RA-LapRep, a simple scaling method that enhances LapRep's ability to reflect state reachability, supported by theoretical analysis and experimental validation.

Findings

RA-LapRep better captures inter-state reachability than LapRep.

Improved reward shaping performance with RA-LapRep.

Enhanced bottleneck state discovery using RA-LapRep.

Abstract

In Reinforcement Learning (RL), Laplacian Representation (LapRep) is a task-agnostic state representation that encodes the geometry of the environment. A desirable property of LapRep stated in prior works is that the Euclidean distance in the LapRep space roughly reflects the reachability between states, which motivates the usage of this distance for reward shaping. However, we find that LapRep does not necessarily have this property in general: two states having small distance under LapRep can actually be far away in the environment. Such mismatch would impede the learning process in reward shaping. To fix this issue, we introduce a Reachability-Aware Laplacian Representation (RA-LapRep), by properly scaling each dimension of LapRep. Despite the simplicity, we demonstrate that RA-LapRep can better capture the inter-state reachability as compared to LapRep, through both theoretical explanations and experimental results. Additionally, we show that this improvement yields a significant boost in reward shaping performance and also benefits bottleneck state discovery.