STO-RL: Offline RL under Sparse Rewards via LLM-Guided Subgoal Temporal Order

TL;DR

STO-RL introduces a novel offline reinforcement learning framework that uses large language models to generate temporally ordered subgoals and applies reward shaping, significantly improving performance on sparse-reward long-horizon tasks.

Contribution

The paper presents a new offline RL approach leveraging LLMs for subgoal ordering and potential-based reward shaping, addressing limitations of existing methods in sparse-reward scenarios.

Findings

Outperforms state-of-the-art offline RL baselines on multiple benchmarks.

Achieves faster convergence and higher success rates.

Demonstrates robustness to noisy LLM-generated subgoals.

Abstract

Offline reinforcement learning (RL) enables policy learning from pre-collected datasets, avoiding costly and risky online interactions, but it often struggles with long-horizon tasks involving sparse rewards. Existing goal-conditioned and hierarchical offline RL methods decompose such tasks and generate intermediate rewards to mitigate limitations of traditional offline RL, but usually overlook temporal dependencies among subgoals and rely on imprecise reward shaping, leading to suboptimal policies. To address these issues, we propose STO-RL (Offline RL using LLM-Guided Subgoal Temporal Order), an offline RL framework that leverages large language models (LLMs) to generate temporally ordered subgoal sequences and corresponding state-to-subgoal-stage mappings. Using this temporal structure, STO-RL applies potential-based reward shaping to transform sparse terminal rewards into dense, temporally consistent signals, promoting subgoal progress while avoiding suboptimal solutions. The resulting augmented dataset with shaped rewards enables efficient offline training of high-performing policies. Evaluations on four discrete and continuous sparse-reward benchmarks demonstrate that STO-RL consistently outperforms state-of-the-art offline goal-conditioned and hierarchical RL baselines, achieving faster convergence, higher success rates, and shorter trajectories. Ablation studies further confirm STO-RL's robustness to imperfect or noisy LLM-generated subgoal sequences, demonstrating that LLM-guided subgoal temporal structures combined with theoretically grounded reward shaping provide a practical and scalable solution for long-horizon offline RL.