Spectral Bellman Method: Unifying Representation and Exploration in RL

TL;DR

The Spectral Bellman Method introduces a spectral framework for representation learning in RL, aligning features with Bellman dynamics to improve exploration and performance in complex tasks.

Contribution

It presents a novel spectral approach based on the Inherent Bellman Error condition, unifying representation and exploration in value-based reinforcement learning.

Findings

Enhanced exploration in hard tasks

Representation learning aligned with Bellman updates

Extension to multi-step Bellman operators

Abstract

Representation learning is critical to the empirical and theoretical success of reinforcement learning. However, many existing methods are induced from model-learning aspects, misaligning them with the RL task in hand. This work introduces the Spectral Bellman Method, a novel framework derived from the Inherent Bellman Error (IBE) condition. It aligns representation learning with the fundamental structure of Bellman updates across a \textit{space} of possible value functions, making it directly suited for value-based RL. Our key insight is a fundamental spectral relationship: under the zero-IBE condition, the transformation of a \textit{distribution} of value functions by the Bellman operator is intrinsically linked to the feature covariance structure. This connection yields a new, theoretically-grounded objective for learning state-action features that capture this Bellman-aligned covariance, requiring only a simple modification to existing algorithms. We demonstrate that our learned representations enable structured exploration by aligning feature covariance with Bellman dynamics, improving performance in hard-exploration and long-horizon tasks. Our framework naturally extends to multi-step Bellman operators, offering a principled path toward learning more powerful and structurally sound representations for value-based RL.