DHP Benchmark: Are LLMs Good NLG Evaluators?

TL;DR

This paper introduces the DHP Benchmark, a framework for quantitatively assessing the evaluation capabilities of large language models across multiple NLG tasks using hierarchical perturbations.

Contribution

It proposes a novel benchmarking framework that systematically measures LLMs' ability to evaluate NLG quality, filling a gap in current assessment methods.

Findings

LLMs show varying strengths across different NLG tasks.

The benchmark reveals limitations in LLMs' evaluation discernment.

Hierarchical perturbation effectively measures evaluation capabilities.

Abstract

Large Language Models (LLMs) are increasingly serving as evaluators in Natural Language Generation (NLG) tasks; this is often referred to as ``LLM-as-a-judge'' paradigm. However, the capabilities of LLMs in evaluating NLG quality remain underexplored. Current studies depend on human assessments and simple metrics that fail to capture the discernment of LLMs across diverse NLG tasks. To address this gap, we propose the Discernment of Hierarchical Perturbation (DHP) benchmarking framework, which provides quantitative discernment scores for LLMs. This framework leverages hierarchically perturbed text data and statistical tests to systematically measure the NLG evaluation capabilities of LLMs. We re-established six evaluation datasets for this benchmark, covering four NLG tasks: Summarization, Story Completion, Question Answering, and Translation. Our comprehensive benchmarking of five major LLM families provides critical insight into their strengths and limitations as NLG evaluators. Our dataset is available at https://huggingface.co/datasets/YCWANGVINCE/DHP_Benchmark.