Complementary Reinforcement Learning

TL;DR

Complementary RL introduces a co-evolving experience extractor and policy actor inspired by neuroscience, significantly improving sample efficiency and performance in reinforcement learning tasks by dynamically leveraging experience during training.

Contribution

The paper proposes a novel Complementary RL framework where experience management co-evolves with the policy, addressing limitations of static experience storage and enhancing learning efficiency.

Findings

Achieves 10% performance improvement over baseline in single-task RL.

Demonstrates robust scalability in multi-task RL settings.

Outperforms outcome-based RL agents that do not learn from experience.

Abstract

Reinforcement Learning (RL) has emerged as a powerful paradigm for training LLM-based agents, yet remains limited by low sample efficiency, stemming not only from sparse outcome feedback but also from the agent's inability to leverage prior experience across episodes. While augmenting agents with historical experience offers a promising remedy, existing approaches suffer from a critical weakness: the experience distilled from history is either stored statically or fail to coevolve with the improving actor, causing a progressive misalignment between the experience and the actor's evolving capability that diminishes its utility over the course of training. Inspired by complementary learning systems in neuroscience, we present Complementary RL to achieve seamless co-evolution of an experience extractor and a policy actor within the RL optimization loop. Specifically, the actor is optimized via sparse outcome-based rewards, while the experience extractor is optimized according to whether its distilled experiences demonstrably contribute to the actor's success, thereby evolving its experience management strategy in lockstep with the actor's growing capabilities. Empirically, Complementary RL outperforms outcome-based agentic RL baselines that do not learn from experience, achieving 10% performance improvement in single-task scenarios and exhibits robust scalability in multi-task settings. These results establish Complementary RL as a paradigm for efficient experience-driven agent learning.