Improve LLM-as-a-Judge Ability as a General Ability

TL;DR

This paper presents a two-stage training approach for large language models to improve their ability as general judges, enhancing accuracy and efficiency in evaluating responses across diverse scenarios, with state-of-the-art results.

Contribution

The work introduces a novel two-stage training method combining supervised fine-tuning and preference optimization, along with an efficient data synthesis technique, to improve LLMs' judging capabilities with less data.

Findings

Achieves state-of-the-art performance on RewardBench.

Requires only 2-40% of data compared to other methods.

Enhances downstream policy optimization through improved judge signals.

Abstract

LLM-as-a-Judge leverages the generative and reasoning capabilities of large language models (LLMs) to evaluate LLM responses across diverse scenarios, providing accurate preference signals. This approach plays a vital role in aligning LLMs with human values, ensuring ethical and reliable AI outputs that align with societal norms. Recent studies have raised many methods to train LLM as generative judges, but most of them are data consuming or lack accuracy, and only focus on LLM's judge ability. In this work, we regard judge ability as a general ability of LLM and implement a two-stage training approach, comprising supervised fine-tuning (SFT) warm-up and direct preference optimization (DPO) enhancement, to achieve judge style adaptation and improve judgment accuracy. Additionally, we introduce an efficient data synthesis method to generate judgmental content. Experimental results demonstrate that our approach, utilizing only about 2% to 40% of the data required by other methods, achieves SOTA performance on RewardBench. Furthermore, our training method enhances the general capabilities of the model by constructing complicated judge task, and the judge signals provided by our model have significantly enhanced the downstream DPO training performance of our internal models in our test to optimize policy model with Judge Model. We also open-source our model weights and training data to facilitate further research.