RTMC: Step-Level Credit Assignment via Rollout Trees

TL;DR

RTMC introduces a critic-free, tree-based advantage estimation method that aggregates return statistics across shared states in rollouts, improving multi-step reinforcement learning performance.

Contribution

The paper proposes RTMC, a novel method that uses rollout trees and state-action signatures for efficient, critic-free advantage estimation in reinforcement learning.

Findings

RTMC improves pass@1 by 3.2 percentage points over GRPO on SWE-bench Verified.

RTMC effectively aggregates return statistics across shared states without learned critics.

The approach enhances multi-step RL credit assignment with lower overhead and increased robustness.

Abstract

Multi-step agentic reinforcement learning benefits from fine-grained credit assignment, yet existing approaches offer limited options: critic-free methods like GRPO assign a uniform advantage to every action in a trajectory, while learned value networks introduce notable overhead and can be fragile under sparse rewards. We observe that group rollouts targeting the same problem often traverse overlapping intermediate states, implicitly forming a tree whose branches diverge at successive decision points. Building on this insight, we introduce Rollout-Tree Monte Carlo (RTMC) advantage estimation, which aggregates return statistics across rollouts sharing a common state to produce per-step Q-values and advantages--without any learned critic. A state-action signature system compresses raw interaction histories into compact, comparable representations, making cross-rollout state matching tractable. On SWE-bench Verified, RTMC improves pass@1 by 3.2 percentage points over GRPO.