Functional Graphs for Predicting and Explaining Goal Failure in Sparse Goal-Conditioned RL

TL;DR

This paper introduces a graph-based analysis of goal-conditioned policies in sparse reinforcement learning, revealing local and global failure modes and providing diagnostic tools for policy success.

Contribution

It proposes a novel local statistic called local goal support (LGS) and a taxonomy of policy-induced graph structures to diagnose goal failures.

Findings

Weak LGS strongly correlates with goal failure across various settings.

Zero LGS exactly prevents goal entry in deterministic GridWorlds.

Local support alone does not guarantee global success due to complex basin structures.

Abstract

Sparse goal-conditioned reinforcement learning can produce policies whose failures are hidden by aggregate success rates. We analyze trained goal-conditioned value policies through the deterministic functional graphs induced by greedy evaluation: for each goal, every state maps to a single successor, decomposing behavior into attractors and basins. This reveals a local-to-global structure in learned policies. We define local goal support (LGS), a one-step statistic measuring the fraction of valid neighboring states whose greedy successor is the goal. In deterministic sparse GridWorlds, zero LGS exactly precludes goal entry from non-goal starts. Empirically, weak LGS is a strong diagnostic of goal-level failure across update rules, curricula, larger grids, and bottleneck geometries: the fixed rule LGS <= 0.5 identifies low-success goals with precision 0.921, recall 0.929, and F1 0.925 in the main 8x8 TD setting, with similar performance across variants. However, local support is not sufficient for global success: some supported goals still fail because distant states are captured by competing attractors or fragmented basin structure. We therefore introduce a compact post-hoc taxonomy of policy-induced graphs -- goal-dominant, competitor-dominated, partial/contested, and fragmented -- to characterize residual failure modes beyond local support. These results show that sparse GCRL failures can be understood as structured policy-induced dynamics, and that local one-step policy structure provides a cheap post-training diagnostic for goal-level failure.