Benchmarking Complex Instruction-Following with Multiple Constraints Composition

TL;DR

ComplexBench is a comprehensive benchmark designed to evaluate large language models' ability to follow complex instructions with multiple constraints, addressing a gap in existing evaluation methods by considering constraint composition.

Contribution

The paper introduces a hierarchical taxonomy and a high-quality dataset for evaluating complex instruction-following, along with an augmented evaluation method for reliability.

Findings

Existing LLMs show significant deficiencies in handling complex instructions with multiple constraints.

ComplexBench effectively identifies limitations in current models' instruction-following capabilities.

The benchmark provides a structured approach to assess the composition of constraints in instructions.

Abstract

Instruction following is one of the fundamental capabilities of large language models (LLMs). As the ability of LLMs is constantly improving, they have been increasingly applied to deal with complex human instructions in real-world scenarios. Therefore, how to evaluate the ability of complex instruction-following of LLMs has become a critical research problem. Existing benchmarks mainly focus on modeling different types of constraints in human instructions while neglecting the composition of different constraints, which is an indispensable constituent in complex instructions. To this end, we propose ComplexBench, a benchmark for comprehensively evaluating the ability of LLMs to follow complex instructions composed of multiple constraints. We propose a hierarchical taxonomy for complex instructions, including 4 constraint types, 19 constraint dimensions, and 4 composition types, and manually collect a high-quality dataset accordingly. To make the evaluation reliable, we augment LLM-based evaluators with rules to effectively verify whether generated texts can satisfy each constraint and composition. Furthermore, we obtain the final evaluation score based on the dependency structure determined by different composition types. ComplexBench identifies significant deficiencies in existing LLMs when dealing with complex instructions with multiple constraints composition.