CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning

TL;DR

CLEANER introduces a self-purification method for agentic RL that uses model-intrinsic self-correction to improve learning efficiency and accuracy, especially in parameter-constrained models, by constructing cleaner training trajectories.

Contribution

The paper presents CLEANER, a novel trajectory purification approach leveraging self-correction to enhance agentic RL training without external filtering or high computational costs.

Findings

Achieves 6%, 3%, and 5% accuracy improvements on benchmarks.

Matches state-of-the-art performance with only one-third of training steps.

Demonstrates scalable, efficient trajectory purification for agentic RL.

Abstract

Agentic Reinforcement Learning (RL) has empowered Large Language Models (LLMs) to utilize tools like Python interpreters for complex problem-solving. However, for parameter-constrained models (e.g., 4B--7B), the exploration phase is often plagued by frequent execution failures, creating noisy trajectories that hinder policy optimization. Under standard outcome-based reward settings, this noise leads to a critical credit assignment issue, where erroneous actions are inadvertently reinforced alongside successful outcomes. Existing mitigations face a dilemma: dense rewards often trigger reward hacking, while supersampling incurs prohibitive computational costs. To address these challenges, we propose CLEANER. Distinct from external filtering methods, CLEANER exploits the model's intrinsic self-correction capabilities to eliminate error-contaminated context directly during data collection. At its core, the Similarity-Aware Adaptive Rollback (SAAR) mechanism autonomously constructs clean, purified trajectories by retrospectively replacing failures with successful self-corrections. Based on semantic similarity, SAAR adaptively regulates replacement granularity from shallow execution repairs to deep reasoning substitutions. By training on these self-purified paths, the model internalizes correct reasoning patterns rather than error-recovery loops. Empirical results on AIME24/25, GPQA, and LiveCodeBench show average accuracy gains of 6%, 3%, and 5% over baselines. Notably, CLEANER matches state-of-the-art performance using only one-third of the training steps, highlighting trajectory purification as a scalable solution for efficient agentic RL. Our models and code are available at GitHub