Impact of Connectivity on Laplacian Representations in Reinforcement Learning

TL;DR

This paper analyzes how the connectivity of the state-graph influences the accuracy of Laplacian-based spectral features in reinforcement learning, providing theoretical bounds and validation through simulations.

Contribution

It establishes bounds on the approximation error of spectral features in RL, linking it to the graph's algebraic connectivity and eigenvector estimation errors.

Findings

Error bounds depend on the graph's algebraic connectivity.

Eigenvector estimation introduces quantifiable error.

Theoretical results validated with gridworld simulations.

Abstract

Learning compact state representations in Markov Decision Processes (MDPs) has proven crucial for addressing the curse of dimensionality in large-scale reinforcement learning (RL) problems. Existing principled approaches leverage structural priors on the MDP by constructing state representations as linear combinations of the state-graph Laplacian eigenvectors. When the transition graph is unknown or the state space is prohibitively large, the graph spectral features can be estimated directly via sample trajectories. In this work, we prove an upper bound on the approximation error of linear value function approximation under the learned spectral features. We show how this error scales with the algebraic connectivity of the state-graph, grounding the approximation quality in the topological structure of the MDP. We further bound the error introduced by the eigenvector estimation itself, leading to an end-to-end error decomposition across the representation learning pipeline. Additionally, our expression of the Laplacian operator for the RL setting, although equivalent to existing ones, prevents some common misunderstandings, of which we show some examples from the literature. Our results hold for general (non-uniform) policies without any assumptions on the symmetry of the induced transition kernel. We validate our theoretical findings with numerical simulations on gridworld environments.