ReVeal: Self-Evolving Code Agents via Reliable Self-Verification

TL;DR

ReVeal is a reinforcement learning framework that enhances code generation by explicitly optimizing self-verification, enabling models to evolve and improve their code through iterative self-assessment and tool-based evaluation, leading to more robust AI agents.

Contribution

ReVeal introduces a novel multi-turn reinforcement learning approach that explicitly optimizes self-verification, co-evolving code and test generation for improved scalability and robustness.

Findings

Enables code evolution over 20+ turns using self-verification.

Significantly improves Pass@k scores, indicating better exploration.

Demonstrates scalability with training on only three datasets.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has advanced the reasoning capabilities of large language models. However, existing methods rely solely on outcome rewards, without explicitly optimizing verification or leveraging reliable signals from realistic environments, leading to unreliable self-verification and limited test-time scaling. To address this, we widen the verification-generation asymmetry by explicitly optimizing self-verification, making it a reliable driver of deeper test-time scaling. We introduce ReVeal, a multi-turn reinforcement learning framework that evolves code generation through self-verification and tool-based evaluation. ReVeal structures long-horizon reasoning as iterative generation-verification turns and incorporates TAPO for turn-level credit assignment, fostering the co-evolution of code and test generation. At inference, this strengthened self-verification enables the model to use self-constructed tests and tool feedback to continuously evolve code for 20+ turns on LiveCodeBench despite training on only three. It also significantly improves Pass@k, indicating stronger exploration that expands the reasoning boundaries of the base model. These findings highlight the promise of ReVeal as a scalable paradigm for RL training and test-time scaling, paving the way for more robust and autonomous AI agents.