Temporally Extended Successor Representations
Matthew J. Sargent, Peter J. Bentley, Caswell Barry, William de Cothi

TL;DR
This paper introduces t-SR, a temporally extended successor representation that captures the dynamics of extended actions, enabling faster policy adaptation in dynamic reward environments without hierarchical learning.
Contribution
t-SR offers a novel approach to temporal abstraction by constructing successor representations over repeated actions, reducing decision complexity without hierarchical policies.
Findings
t-SR adapts policies faster in sparse reward gridworlds.
t-SR requires fewer policy samples than non-temporally extended methods.
t-SR effectively handles environments with dynamic reward structures.
Abstract
We present a temporally extended variation of the successor representation, which we term t-SR. t-SR captures the expected state transition dynamics of temporally extended actions by constructing successor representations over primitive action repeats. This form of temporal abstraction does not learn a top-down hierarchy of pertinent task structures, but rather a bottom-up composition of coupled actions and action repetitions. This lessens the amount of decisions required in control without learning a hierarchical policy. As such, t-SR directly considers the time horizon of temporally extended action sequences without the need for predefined or domain-specific options. We show that in environments with dynamic reward structure, t-SR is able to leverage both the flexibility of the successor representation and the abstraction afforded by temporally extended actions. Thus, in a series of…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsReinforcement Learning in Robotics · Neural dynamics and brain function · Explainable Artificial Intelligence (XAI)
