Rewarding the Scientific Process: Process-Level Reward Modeling for Agentic Data Analysis

TL;DR

This paper introduces DataPRM, a process-level reward model for data analysis agents that detects silent errors and improves policy learning, outperforming baselines with only 4B parameters.

Contribution

The work presents DataPRM, a novel environment-aware generative reward model that actively verifies intermediate states and distinguishes error types, advancing process reward modeling in dynamic data analysis.

Findings

DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench.

DataPRM achieves 11.28% improvement on DABStep with Best-of-N inference.

DataPRM outperforms strong baselines with only 4B parameters and generalizes across strategies.

Abstract

Process Reward Models (PRMs) have achieved remarkable success in augmenting the reasoning capabilities of Large Language Models (LLMs) within static domains such as mathematics. However, their potential in dynamic data analysis tasks remains underexplored. In this work, we first present a empirical study revealing that general-domain PRMs struggle to supervise data analysis agents. Specifically, they fail to detect silent errors, logical flaws that yield incorrect results without triggering interpreter exceptions, and erroneously penalize exploratory actions, mistaking necessary trial-and-error exploration for grounding failures. To bridge this gap, we introduce DataPRM, a novel environment-aware generative process reward model that (1) can serve as an active verifier, autonomously interacting with the environment to probe intermediate execution states and uncover silent errors, and (2) employs a reflection-aware ternary reward strategy that distinguishes between correctable grounding errors and irrecoverable mistakes. We design a scalable pipeline to construct over 8K high-quality training instances for DataPRM via diversity-driven trajectory generation and knowledge-augmented step-level annotation. Experimental results demonstrate that DataPRM improves downstream policy LLMs by 7.21% on ScienceAgentBench and 11.28% on DABStep using Best-of-N inference. Notably, with only 4B parameters, DataPRM outperforms strong baselines, and exhibits robust generalizability across diverse Test-Time Scaling strategies. Furthermore, integrating DataPRM into Reinforcement Learning yields substantial gains over outcome-reward baselines, achieving 78.73% on DABench and 64.84% on TableBench, validating the effectiveness of process reward supervision. Code is available at https://github.com/zjunlp/DataMind.