Hierarchical Successor Representation for Robust Transfer

TL;DR

This paper introduces Hierarchical Successor Representation (HSR), a novel approach that incorporates temporal abstractions to create stable, interpretable, and transfer-efficient predictive representations in complex environments.

Contribution

The paper proposes HSR, a hierarchical extension of SR that improves robustness to policy changes and environmental complexity, enabling better transfer and exploration.

Findings

HSR learns stable, policy-agnostic features.

NMF on HSR yields sparse, interpretable maps.

HSR enhances transfer and exploration in complex environments.

Abstract

The successor representation (SR) provides a powerful framework for decoupling predictive dynamics from rewards, enabling rapid generalisation across reward configurations. However, the classical SR is limited by its inherent policy dependence: policies change due to ongoing learning, environmental non-stationarities, and changes in task demands, making established predictive representations obsolete. Furthermore, in topologically complex environments, SRs suffer from spectral diffusion, leading to dense and overlapping features that scale poorly. Here we propose the Hierarchical Successor Representation (HSR) for overcoming these limitations. By incorporating temporal abstractions into the construction of predictive representations, HSR learns stable state features which are robust to task-induced policy changes. Applying non-negative matrix factorisation (NMF) to the HSR yields a sparse, low-rank state representation that facilitates highly sample-efficient transfer to novel tasks in multi-compartmental environments. Further analysis reveals that HSR-NMF discovers interpretable topological structures, providing a policy-agnostic hierarchical map that effectively bridges model-free optimality and model-based flexibility. Beyond providing a useful basis for task-transfer, we show that HSR's temporally extended predictive structure can also be leveraged to drive efficient exploration, effectively scaling to large, procedurally generated environments.