In-Context Learning User Simulators for Task-Oriented Dialog Systems

TL;DR

This paper introduces a new in-context learning approach using large language models to create user simulators for task-oriented dialog systems, reducing the need for extensive data and rules, and providing insights into system errors.

Contribution

It proposes a novel, data-efficient user simulation method leveraging large language models and in-context learning, improving flexibility and reducing manual effort.

Findings

Generated diverse user utterances based on goals and limited examples

Eliminated need for rule-based or annotated data

Identified common errors in dialog interactions

Abstract

This paper presents a novel application of large language models in user simulation for task-oriented dialog systems, specifically focusing on an in-context learning approach. By harnessing the power of these models, the proposed approach generates diverse utterances based on user goals and limited dialog examples. Unlike traditional simulators, this method eliminates the need for labor-intensive rule definition or extensive annotated data, making it more efficient and accessible. Additionally, an error analysis of the interaction between the user simulator and dialog system uncovers common mistakes, providing valuable insights into areas that require improvement. Our implementation is available at https://github.com/telepathylabsai/prompt-based-user-simulator.