RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents

TL;DR

RLVMR introduces a reinforcement learning framework that rewards explicit, verifiable reasoning processes, significantly improving robustness and efficiency in long-horizon tasks by enhancing reasoning quality and reducing redundant actions.

Contribution

The paper presents RLVMR, a novel RL framework that incorporates process-level supervision and rule-based rewards for better reasoning in long-horizon tasks, achieving state-of-the-art results.

Findings

Achieved 83.6% success on challenging unseen tasks.

Reduced redundant actions and improved error recovery.

Enhanced reasoning quality and robustness of agents.

Abstract

The development of autonomous agents for complex, long-horizon tasks is a central goal in AI. However, dominant training paradigms face a critical limitation: reinforcement learning (RL) methods that optimize solely for final task success often reinforce flawed or inefficient reasoning paths, a problem we term inefficient exploration. This leads to agents that are brittle and fail to generalize, as they learn to find solutions without learning how to reason coherently. To address this, we introduce RLVMR, a novel framework that integrates dense, process-level supervision into end-to-end RL by rewarding verifiable, meta-reasoning behaviors. RLVMR equips an agent to explicitly tag its cognitive steps, such as planning, exploration, and reflection, and provides programmatic, rule-based rewards for actions that contribute to effective problem-solving. These process-centric rewards are combined with the final outcome signal and optimized using a critic-free policy gradient method. On the challenging ALFWorld and ScienceWorld benchmarks, RLVMR achieves new state-of-the-art results, with our 7B model reaching an 83.6% success rate on the most difficult unseen task split. Our analysis confirms these gains stem from improved reasoning quality, including significant reductions in redundant actions and enhanced error recovery, leading to more robust, efficient, and interpretable agents.