Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback

TL;DR

This paper presents RealTOD, a framework that enhances multi-turn task-oriented dialog systems by using prompt chaining and detailed feedback to improve API call accuracy and generalization across domains.

Contribution

The paper introduces RealTOD, a novel approach combining prompt chaining and fine-grained feedback to significantly improve multi-turn task completion in dialog systems.

Findings

RealTOD surpasses state-of-the-art API call accuracy on SGD by 37.10%.

RealTOD outperforms supervised baseline SimpleTOD by 10.32% on BiTOD.

Human evaluations show improved task completion, fluency, and informativeness.

Abstract

Task-oriented dialog (TOD) systems facilitate users in accomplishing complex, multi-turn tasks through natural language. While instruction-tuned large language models (LLMs) have demonstrated strong performance on a range of single-turn NLP tasks, they often struggle with reliable multi-turn task completion in TOD settings, particularly when generating API calls required to interact with external systems. To address this, we introduce RealTOD, a novel framework that improves LLM-based TOD systems through (1) prompt chaining and (2) fine-grained feedback. Prompt chaining enables zero-shot generalization to new domains by automatically synthesizing a schema-aligned in-context example for the target task. Fine-grained feedback verifies each generated API call against the domain schema, identifies specific errors, and provides targeted correction prompts. To evaluate task completion reliability, we introduce full API Call Accuracy as a robust metric, along with detailed sub-metrics to capture common failure modes. We conduct extensive experiments on the SGD and BiTOD benchmarks using four LLMs. RealTOD improves Full API accuracy, surpassing state-of-the-art AutoTOD by 37.10% on SGD and supervised learning-based baseline SimpleTOD by 10.32% on BiTOD. Human evaluations further confirm that LLMs integrated with RealTOD achieve superior task completion, fluency, and informativeness compared to existing methods.