Subgoal Graph-Augmented Planning for LLM-Guided Open-World Reinforcement Learning

TL;DR

This paper introduces SGA-ACR, a framework that enhances LLM-guided reinforcement learning by integrating environment-specific subgoal graphs and a multi-LLM pipeline to improve plan feasibility, verification, and execution in open-world tasks.

Contribution

It proposes a novel subgoal graph-augmented planning framework with explicit generation, critique, and refinement stages for better alignment between plans and environment actions.

Findings

Improved subgoal feasibility and reliability in open-world tasks.

Enhanced alignment between high-level plans and environment-specific actions.

Demonstrated effectiveness on 22 diverse tasks in the Crafter game.

Abstract

Large language models (LLMs) offer strong high-level planning capabilities for reinforcement learning (RL) by decomposing tasks into subgoals. However, their practical utility is limited by poor planning-execution alignment, which reflects a critical gap between abstract plans and actionable, environment-compatible behaviors. This misalignment arises from two interrelated limitations: (1) LLMs often produce subgoals that are semantically plausible but infeasible or irrelevant in the target environment due to insufficient grounding in environment-specific knowledge, and (2) single-LLM planning conflates generation with self-verification, resulting in overconfident yet unreliable subgoals that frequently fail during execution. To address these challenges, we propose Subgoal Graph-Augmented Actor-Critic-Refiner (SGA-ACR), a framework that integrates an environment-specific subgoal graph and structured entity knowledge with a multi-LLM planning pipeline that explicitly separates generation, critique, and refinement to produce executable and verifiable subgoals. A subgoal tracker further monitors execution progress, provides auxiliary rewards, and adaptively updates the subgoal graph to maintain alignment between plans and actions. Experimental results on 22 diverse tasks in the open-world game "Crafter" demonstrate the effectiveness of our proposed method.