StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following

TL;DR

StructFlowBench introduces a new benchmark for evaluating multi-turn instruction following in language models, emphasizing the importance of structural dependencies between dialogue turns, revealing current models' deficiencies in understanding these structures.

Contribution

This work presents the first benchmark focusing on structural flow modeling in multi-turn dialogues, with a framework of six inter-turn relationships for evaluation and customization.

Findings

Current models show significant gaps in understanding dialogue structures.

The benchmark enables tailored dialogue flow generation for specific scenarios.

Systematic evaluation of 13 LLMs highlights deficiencies in multi-turn comprehension.

Abstract

Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependencies between dialogue turns that distinguish multi-turn from single-turn interactions. These structural dependencies not only reflect user intent but also establish an essential second dimension for the instruction following evaluation beyond constraint satisfaction. To address this gap, we propose StructFlowBench, a multi-turn instruction following benchmark with structural flow modeling. The benchmark defines an innovative structural flow framework with six fundamental inter-turn relationships. These relationships introduce novel structural constraints for model evaluation and also serve as generation parameters for creating customized dialogue flows tailored to specific scenarios. Adopting established LLM-based automatic evaluation methodologies, we conduct systematic evaluations of 13 leading open-source and closed-source LLMs. Experimental results reveal significant deficiencies in current models' comprehension of multi-turn dialogue structures. The code is available at https://github.com/MLGroupJLU/StructFlowBench.