Action-Sufficient Goal Representations

TL;DR

This paper introduces an information-theoretic framework for goal representations in hierarchical offline goal-conditioned reinforcement learning, emphasizing action sufficiency over value sufficiency to improve control performance.

Contribution

It defines action sufficiency, proves its distinction from value sufficiency, and shows that standard training naturally induces action-sufficient representations for better control.

Findings

Action-sufficient representations outperform value-sufficient ones in control tasks.

Standard log-loss training induces action-sufficient goal representations.

Empirical validation on benchmark demonstrates improved control success.

Abstract

Hierarchical policies in offline goal-conditioned reinforcement learning (GCRL) addresses long-horizon tasks by decomposing control into high-level subgoal planning and low-level action execution. A critical design choice in such architectures is the goal representation-the compressed encoding of goals that serves as the interface between these levels. Existing approaches commonly derive goal representations while learning value functions, implicitly assuming that preserving information sufficient for value estimation is adequate for optimal control. We show that this assumption can fail, even when the value estimation is exact, as such representations may collapse goal states that need to be differentiated for action learning. To address this, we introduce an information-theoretic framework that defines action sufficiency, a condition on goal representations necessary for optimal action selection. We prove that value sufficiency does not imply action sufficiency and empirically verify that the latter is more strongly associated with control success in a discrete environment. We further demonstrate that standard log-loss training of low-level policies naturally induces action-sufficient representations. Our experimental results a popular benchmark demonstrate that our actor-derived representations consistently outperform representations learned via value estimation.