LLM-Driven Multi-Turn Task-Oriented Dialogue Synthesis for Realistic Reasoning

TL;DR

This paper introduces a novel LLM-driven framework for synthesizing multi-turn, task-oriented dialogues that reflect real-world reasoning scenarios, aiming to improve the evaluation and development of LLM reasoning abilities.

Contribution

It presents a trilevel optimization-based method for generating realistic, coherent dialogues and reasoning tasks, addressing limitations of existing benchmarks and data contamination issues.

Findings

Synthetic datasets introduce complex reasoning challenges.

Generated dialogues are contextually coherent and realistic.

Method enhances LLM reasoning evaluation and training.

Abstract

The reasoning capability of large language models (LLMs), defined as their ability to analyze, infer, and make decisions based on input information, is essential for building intelligent task-oriented dialogue systems. However, existing benchmarks do not sufficiently reflect the complexity of real-world scenarios, which limits their effectiveness in evaluating and enhancing LLM reasoning in practical contexts. Many current reasoning datasets are overly simplistic and abstract, often disconnected from realistic task flows, domain constraints, and operational rules, making it difficult to effectively evaluate LLMs' logical reasoning ability. In addition, data contamination from pretraining corpora undermines the reliability of evaluation results, and traditional crowdsourcing methods for dataset construction are labor-intensive and difficult to scale. To address these challenges, we propose a LLM-driven framework for synthesizing multi-turn, task-oriented dialogues grounded in realistic reasoning scenarios, leveraging trilevel optimization to enhance dialogue quality. Our method generates dialogues grounded in authentic task scenarios, enriched with real-world information, and exhibiting strong contextual coherence. Corresponding reasoning tasks are carefully designed around these dialogues and iteratively refined to continuously improve the tasks' quality and challenge. The resulting dataset serves as a valuable benchmark for assessing and advancing the realistic logical reasoning capabilities of LLMs. Experimental results show that our synthetic data-based reasoning tasks introduce non-trivial reasoning challenges and provide meaningful support for improving the reasoning capabilities of LLMs.