J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning

TL;DR

This paper introduces J1, a reinforcement learning framework that trains large language models to improve their evaluation capabilities by teaching them to think before judging, leading to state-of-the-art performance on multiple benchmarks.

Contribution

The paper presents a novel RL-based method for training LLM judges to think before judging, converting tasks into a unified format with verifiable rewards, and demonstrates superior performance with synthetic data training.

Findings

J1 models achieve state-of-the-art results on multiple benchmarks.

J1 outperforms larger models trained on real data using synthetic data.

Qualitative analysis shows J1 develops systematic evaluation strategies.

Abstract

The progress of AI is bottlenecked by the quality of evaluation, making powerful LLM-as-a-Judge models a core solution. The efficacy of these judges depends on their chain-of-thought reasoning, creating a critical need for methods that can effectively optimize this reasoning process. In this work, we introduce J1, a reinforcement learning framework for teaching LLM judges to think before making decisions. Our core contribution lies in converting all judgment tasks for non-verifiable and verifiable prompts into a unified format with verifiable rewards, enabling direct optimization of evaluation quality while mitigating positional bias. We then use RL to train thinking-judges at scales of 8B, 32B, and 70B and show that they obtain state-of-the-art performance across multiple benchmarks. In particular, J1-Qwen-32B, our multitasked pointwise and pairwise judge also outperforms o1-mini, o3, and a much larger 671B DeepSeek-R1 on some benchmarks, while only training on synthetic data. Through comprehensive ablations of pairwise, pointwise, and multitask J1 variants, we demonstrate the effectiveness of our approach across seed prompts, reward strategies, and training recipes. Qualitative analysis reveals that J1 develops systematic evaluation strategies, including dynamic criteria generation, reference answer creation, iterative self-correction of initial assessments, and feedback generation for low-quality responses.