HiMAC: Hierarchical Macro-Micro Learning for Long-Horizon LLM Agents

TL;DR

HiMAC introduces a hierarchical reinforcement learning framework for LLM agents, decomposing long-horizon tasks into planning and execution, leading to improved performance and sample efficiency in complex environments.

Contribution

The paper presents a novel hierarchical RL approach with a critic-free training paradigm and iterative co-evolution strategy for better long-horizon decision-making in LLM agents.

Findings

HiMAC outperforms baselines on ALFWorld, WebShop, and Sokoban.

Achieves state-of-the-art performance and better sample efficiency.

Structured hierarchy is crucial for robust long-horizon agentic intelligence.

Abstract

Large language model (LLM) agents have recently demonstrated strong capabilities in interactive decision-making, yet they remain fundamentally limited in long-horizon tasks that require structured planning and reliable execution. Existing approaches predominantly rely on flat autoregressive policies, where high-level reasoning and low-level actions are generated within a single token sequence, leading to inefficient exploration and severe error propagation over extended trajectories. In this work, we propose HiMAC, a hierarchical agentic RL framework that explicitly decomposes long-horizon decision-making into macro-level planning and micro-level execution. HiMAC models reasoning as a structured blueprint generation process followed by goal-conditioned action execution, enabling robust long-horizon planning within LLM-based agents. To train this hierarchy efficiently, we introduce a critic-free hierarchical policy optimization paradigm that extends group-based reinforcement learning to bi-level structures through hierarchical relative advantage estimation. Furthermore, we propose an iterative co-evolution training strategy that alternates between planner exploration and executor adaptation, mitigating the non-stationarity inherent in hierarchical learning. Extensive experiments on ALFWorld, WebShop, and Sokoban demonstrate that HiMAC consistently outperforms strong prompting and reinforcement learning baselines, achieving state-of-the-art performance and substantially improved sample efficiency across both text-based and visually grounded environments. Our results show that introducing structured hierarchy, rather than increasing model scale alone, is a key factor for enabling robust long-horizon agentic intelligence.