Spectral Representation-based Reinforcement Learning

TL;DR

This paper introduces spectral representations for reinforcement learning, providing a theoretically grounded, efficient approach to model dynamics and improve policy optimization in large state-action spaces.

Contribution

It proposes a spectral decomposition framework for RL that enhances understanding, stability, and efficiency, with methods applicable to various transition structures and partial observability.

Findings

Achieves comparable or superior performance to state-of-the-art methods on DeepMind Control Suite tasks.

Provides a theoretical characterization of spectral representations for RL.

Develops practical algorithms based on spectral decomposition for different transition structures.

Abstract

In real-world applications with large state and action spaces, reinforcement learning (RL) typically employs function approximations to represent core components like the policies, value functions, and dynamics models. Although powerful approximations such as neural networks offer great expressiveness, they often present theoretical ambiguities, suffer from optimization instability and exploration difficulty, and incur substantial computational costs in practice. In this paper, we introduce the perspective of spectral representations as a solution to address these difficulties in RL. Stemming from the spectral decomposition of the transition operator, this framework yields an effective abstraction of the system dynamics for subsequent policy optimization while also providing a clear theoretical characterization. We reveal how to construct spectral representations for transition operators that possess latent variable structures or energy-based structures, which implies different learning methods to extract spectral representations from data. Notably, each of these learning methods realizes an effective RL algorithm under this framework. We also provably extend this spectral view to partially observable MDPs. Finally, we validate these algorithms on over 20 challenging tasks from the DeepMind Control Suite, where they achieve performances comparable or superior to current state-of-the-art model-free and model-based baselines.