Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

TL;DR

This paper introduces TIR-Judge, an RL framework that enhances LLM judges with tool-integrated reasoning, enabling more accurate evaluations across diverse domains and outperforming existing models on multiple benchmarks.

Contribution

The paper presents a novel RL-based training method for LLM judges that incorporates a code executor, improving their ability to verify complex constraints without relying on distillation.

Findings

TIR-Judge outperforms strong reasoning-based judges by up to 7.7%.

TIR-Judge-Zero matches the performance of distilled variants without using distilled data.

Listwise performance of TIR-Judge is comparable to larger models like Claude-Opus-4.

Abstract

Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.