CoDA: A Context-Decoupled Hierarchical Agent with Reinforcement Learning

TL;DR

CoDA introduces a hierarchical reinforcement learning framework that decouples planning and execution in LLM agents, effectively mitigating context overload and improving performance on complex multi-hop tasks.

Contribution

It proposes a novel hierarchical RL framework with context decoupling and a joint training method, enhancing LLM agent robustness and performance in long-context scenarios.

Findings

Significant performance gains on multi-hop question-answering benchmarks.

Robustness in long-context scenarios where other models degrade.

Effective mitigation of context explosion through hierarchical design.

Abstract

Large Language Model (LLM) agents trained with reinforcement learning (RL) show great promise for solving complex, multi-step tasks. However, their performance is often crippled by "Context Explosion", where the accumulation of long text outputs overwhelms the model's context window and leads to reasoning failures. To address this, we introduce CoDA, a Context-Decoupled hierarchical Agent, a simple but effective reinforcement learning framework that decouples high-level planning from low-level execution. It employs a single, shared LLM backbone that learns to operate in two distinct, contextually isolated roles: a high-level Planner that decomposes tasks within a concise strategic context, and a low-level Executor that handles tool interactions in an ephemeral, isolated workspace. We train this unified agent end-to-end using PECO (Planner-Executor Co-Optimization), a reinforcement learning methodology that applies a trajectory-level reward to jointly optimize both roles, fostering seamless collaboration through context-dependent policy updates. Extensive experiments demonstrate that CoDA achieves significant performance improvements over state-of-the-art baselines on complex multi-hop question-answering benchmarks, and it exhibits strong robustness in long-context scenarios, maintaining stable performance while all other baselines suffer severe degradation, thus further validating the effectiveness of our hierarchical design in mitigating context overload.