Cochain Perspectives on Temporal-Difference Signals for Learning Beyond Markov Dynamics

TL;DR

This paper introduces a topological perspective on temporal-difference learning in reinforcement learning, addressing non-Markovian dynamics by decomposing TD errors and proposing a new algorithm that improves performance in complex environments.

Contribution

It presents a novel topological framework for analyzing TD errors, including a Hodge-type decomposition and a new algorithm, HodgeFlow Policy Search, for better learning under non-Markovian dynamics.

Findings

HFPS significantly improves RL performance in non-Markovian environments.

The topological decomposition offers new insights into TD errors and dynamics.

The approach provides stability and sensitivity guarantees for RL algorithms.

Abstract

Non-Markovian dynamics are commonly found in real-world environments due to long-range dependencies, partial observability, and memory effects. The Bellman equation that is the central pillar of Reinforcement learning (RL) becomes only approximately valid under Non-Markovian. Existing work often focus on practical algorithm designs and offer limited theoretical treatment to address key questions, such as what dynamics are indeed capturable by the Bellman framework and how to inspire new algorithm classes with optimal approximations. In this paper, we present a novel topological viewpoint on temporal-difference (TD) based RL. We show that TD errors can be viewed as 1-cochain in the topological space of state transitions, while Markov dynamics are then interpreted as topological integrability. This novel view enables us to obtain a Hodge-type decomposition of TD errors into an integrable component and a topological residual, through a Bellman-de Rham projection. We further propose HodgeFlow Policy Search (HFPS) by fitting a potential network to minimize the non-integrable projection residual in RL, achieving stability/sensitivity guarantees. In numerical evaluations, HFPS is shown to significantly improve RL performance under non-Markovian.