TOD-ProcBench: Benchmarking Complex Instruction-Following in Task-Oriented Dialogues

TL;DR

This paper introduces TOD-ProcBench, a comprehensive benchmark for evaluating large language models' ability to understand and follow complex, multi-level instructions in task-oriented dialogues, addressing limitations of previous simplified benchmarks.

Contribution

We propose TOD-ProcBench, a challenging, multi-task benchmark with complex instructions and constraints, designed to systematically assess LLMs' instruction-following in multi-turn dialogues.

Findings

LLMs show varied performance across tasks

Multilingual settings impact instruction compliance

Instruction text format influences model understanding

Abstract

In real-world task-oriented dialogue (TOD) settings, agents are required to strictly adhere to complex instructions while conducting multi-turn conversations with customers. These instructions are typically presented in natural language format and include general guidelines and step-by-step procedures with complex constraints. Existing TOD benchmarks often oversimplify the complex nature of these instructions by reducing them to simple schemas composed of intents, slots, and API call configurations. To address this gap and systematically benchmark LLMs' instruction-following capabilities, we propose TOD-ProcBench, a challenging benchmark featuring complex process instructions with intricate, fine-grained constraints that evaluates various LLMs' abilities to understand and follow instructions in multi-turn TODs. Our benchmark dataset comprises instruction documents derived from the high-quality ABCD dataset with corresponding conversations under human quality control. We formulate fine-grained constraints and action procedures as multi-level condition-action instruction statements. We design three tasks to comprehensively benchmark LLMs' complex instruction-following capabilities in multi-turn TODs. Task 1 evaluates how LLMs retrieve the most relevant statement from a complex instruction and predict the corresponding next action. In Task 2, we synthesize instruction-violating responses by injecting inconsistencies and manipulating the original instructions, and then we analyze how effectively LLMs can identify instruction-violating responses. Task 3 investigates LLMs' abilities in conditional generation of instruction-following responses based on the original complex instructions. Additionally, we conduct studies on the impact of multilingual settings and different instruction text formats on compliance performance. We release our benchmark under the Llama 3.3 Community License Agreement.