Towards Better Evaluation of Instruction-Following: A Case-Study in Summarization

TL;DR

This paper critically assesses existing evaluation metrics for instruction-following in large language models, introduces a new dataset for summarization, and proposes improved LLM-based evaluation methods that match the performance of traditional reference-based metrics.

Contribution

It provides a comprehensive meta-evaluation of current metrics, introduces the riSum dataset for grounded summarization, and develops new LLM-based evaluation methods that are reference-free and highly effective.

Findings

Evaluation metrics show varying agreement with human judgment.

The riSum dataset enables better assessment of instruction-following.

Proposed LLM-based metrics match the performance of reference-based methods.

Abstract

Despite recent advances, evaluating how well large language models (LLMs) follow user instructions remains an open problem. While evaluation methods of language models have seen a rise in prompt-based approaches, limited work on the correctness of these methods has been conducted. In this work, we perform a meta-evaluation of a variety of metrics to quantify how accurately they measure the instruction-following abilities of LLMs. Our investigation is performed on grounded query-based summarization by collecting a new short-form, real-world dataset riSum, containing 300 document-instruction pairs with 3 answers each. All 900 answers are rated by 3 human annotators. Using riSum, we analyze the agreement between evaluation methods and human judgment. Finally, we propose new LLM-based reference-free evaluation methods that improve upon established baselines and perform on par with costly reference-based metrics that require high-quality summaries.