Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems

TL;DR

This paper presents DAUS, a domain-aware user simulator based on large language models, which improves the realism and goal fulfillment in task-oriented dialogue systems by fine-tuning on real dialogue data.

Contribution

Introduces DAUS, a novel LLM-based user simulator that is fine-tuned on real dialogues to enhance coherence and reduce hallucinations in task-oriented systems.

Findings

Significant improvements in user goal fulfillment on benchmarks

Fine-tuning enhances coherence with user goals

Mitigates hallucinations in simulator responses

Abstract

In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses.