ToolRM: Towards Agentic Tool-Use Reward Modeling

TL;DR

ToolRM introduces specialized reward models for tool-use tasks in LLMs, improving reward judgment accuracy and enabling scalable, efficient agentic AI applications with broad evaluation and practical utility.

Contribution

The paper presents a novel pipeline for creating high-quality preference data and introduces ToolRM, a family of reward models tailored for tool-use scenarios in LLMs, with extensive evaluation benchmarks.

Findings

ToolRM models achieve up to 17.94% higher accuracy in reward judgments.

Generative ToolRM generalizes to critique tasks like self-correction.

Inference-time scaling reduces output token usage by over 66%.

Abstract

Reward models (RMs) play a critical role in aligning large language models (LLMs) with human preferences. Yet in the domain of tool learning, the lack of RMs specifically designed for function-calling tasks has limited progress toward more capable agentic AI. We introduce ToolRM, a family of lightweight reward models tailored for general tool-use scenarios. To build these models, we propose a novel pipeline that constructs high-quality pairwise preference data using rule-based scoring and multidimensional sampling. This yields ToolPref-Pairwise-30K, a diverse, balanced, and challenging preference dataset that supports both generative and discriminative reward modeling. We also introduce TRBench$_{BFCL}$, a benchmark built on the agent evaluation suite BFCL to evaluate RMs on tool calling tasks. Trained on our constructed data, models from the Qwen3-4B/8B series achieve up to 17.94% higher accuracy, substantially outperforming frontier LLMs and RMs in pairwise reward judgments. Beyond training objectives, generative ToolRM generalizes to broader critique tasks, including Best-of-N sampling and self-correction. Experiments on ACEBench highlight its effectiveness and efficiency, enabling inference-time scaling while reducing output token usage by over 66%. Its support for downstream RL training further validates its practical utility. We release data to facilitate future research.