Large Language Models as User-Agents for Evaluating Task-Oriented-Dialogue Systems

TL;DR

This paper explores using large language models as context-aware user-agents to evaluate task-oriented dialogue systems more effectively than traditional offline datasets, emphasizing improved diversity and task completion metrics.

Contribution

It introduces a novel framework leveraging LLM-based user-agents for dynamic evaluation of TOD systems, including methodologies for automatic assessment.

Findings

Enhanced diversity in user-agent interactions

Improved task completion metrics with better prompts

Proposed automatic evaluation methodologies

Abstract

Traditionally, offline datasets have been used to evaluate task-oriented dialogue (TOD) models. These datasets lack context awareness, making them suboptimal benchmarks for conversational systems. In contrast, user-agents, which are context-aware, can simulate the variability and unpredictability of human conversations, making them better alternatives as evaluators. Prior research has utilized large language models (LLMs) to develop user-agents. Our work builds upon this by using LLMs to create user-agents for the evaluation of TOD systems. This involves prompting an LLM, using in-context examples as guidance, and tracking the user-goal state. Our evaluation of diversity and task completion metrics for the user-agents shows improved performance with the use of better prompts. Additionally, we propose methodologies for the automatic evaluation of TOD models within this dynamic framework.