FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models

TL;DR

FollowEval is a comprehensive, multi-dimensional benchmark in English and Chinese, crafted by experts, to evaluate large language models' instruction-following abilities across five key areas, revealing significant performance gaps.

Contribution

This paper introduces FollowEval, a novel benchmark with human-crafted multilingual test examples covering multiple instruction-following dimensions.

Findings

LLMs perform significantly worse than humans on FollowEval

Benchmark covers five critical instruction-following dimensions

Includes multilingual test examples in English and Chinese

Abstract

The effective assessment of the instruction-following ability of large language models (LLMs) is of paramount importance. A model that cannot adhere to human instructions might be not able to provide reliable and helpful responses. In pursuit of this goal, various benchmarks have been constructed to evaluate the instruction-following capacity of these models. However, these benchmarks are limited to a single language and are constructed using automated approaches, which restricts their applicability and the quality of the test examples they contain. To bridge this gap, we introduce the FollowEval benchmark in this paper. This benchmark is composed of instances in both English and Chinese, and all test examples are crafted by human experts. Furthermore, the FollowEval benchmark is designed to assess LLMs across five critical dimensions of instruction following: string manipulation, commonsense reasoning, logical reasoning, spatial reasoning, and response constraints. To enhance the complexity and present a sufficient challenge, each test example is designed to evaluate more than one dimension. We have evaluated various LLMs using the FollowEval benchmark and found that their performance significantly lags behind that of humans. This highlights the considerable room for improvement in the instruction-following ability of these models.