Instruction-Following Evaluation for Large Language Models

TL;DR

This paper introduces IFEval, a standardized, reproducible benchmark for evaluating large language models' ability to follow natural language instructions, addressing limitations of human and LLM-based evaluations.

Contribution

The paper presents IFEval, a new benchmark with verifiable instructions and a diverse set of prompts, enabling consistent and objective evaluation of LLM instruction-following capabilities.

Findings

Two widely available LLMs evaluated on IFEval.

IFEval provides a reproducible and objective assessment.

Benchmark covers 25 instruction types with 500 prompts.

Abstract

One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of "verifiable instructions" such as "write in more than 400 words" and "mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval