On Evaluating LLM Alignment by Evaluating LLMs as Judges

TL;DR

This paper explores how large language models (LLMs) can be evaluated for alignment with human preferences by analyzing their evaluation capabilities, proposing a new benchmark that assesses alignment through evaluation performance rather than direct output assessment.

Contribution

It introduces AlignEval, a benchmark that measures LLM alignment by evaluating their judging ability, revealing a strong correlation between generation and evaluation capabilities of LLMs.

Findings

AlignEval matches or exceeds existing benchmarks in capturing human preferences.

Strong correlation found between LLMs' generation and evaluation capabilities.

Proposes a new paradigm for assessing LLM alignment without direct output evaluation.

Abstract

Alignment with human preferences is an important evaluation aspect of LLMs, requiring them to be helpful, honest, safe, and to precisely follow human instructions. Evaluating large language models' (LLMs) alignment typically involves directly assessing their open-ended responses, requiring human annotators or strong LLM judges. Conversely, LLMs themselves have also been extensively evaluated as judges for assessing alignment. In this work, we examine the relationship between LLMs' generation and evaluation capabilities in aligning with human preferences. To this end, we first conduct a comprehensive analysis of the generation-evaluation consistency (GE-consistency) among various LLMs, revealing a strong correlation between their generation and evaluation capabilities when evaluated by a strong LLM preference oracle. Utilizing this finding, we propose a benchmarking paradigm that measures LLM alignment with human preferences without directly evaluating their generated outputs, instead assessing LLMs in their role as evaluators. Our evaluation shows that our proposed benchmark, AlignEval, matches or surpasses widely used automatic LLM evaluation benchmarks, such as AlpacaEval and Arena-Hard, in capturing human preferences when ranking LLMs. Our study offers valuable insights into the connection between LLMs' generation and evaluation capabilities, and introduces a benchmark that assesses alignment without directly evaluating model outputs.