Beyond Single-User Dialogue: Assessing Multi-User Dialogue State Tracking Capabilities of Large Language Models

TL;DR

This paper evaluates large language models' ability to track dialogue states in multi-user conversations, revealing significant performance challenges and emphasizing the need for improved models in realistic multi-party interactions.

Contribution

It introduces a novel methodology for assessing multi-user dialogue state tracking in LLMs using extended datasets and speech act theory-based utterance generation.

Findings

LLMs show a significant performance drop in multi-user DST compared to single-user scenarios.

The study highlights current limitations of LLMs in multi-party dialogue understanding.

Results suggest the necessity for developing specialized models for multi-user dialogue tracking.

Abstract

Large language models (LLMs) have demonstrated remarkable performance in zero-shot dialogue state tracking (DST), reducing the need for task-specific training. However, conventional DST benchmarks primarily focus on structured user-agent conversations, failing to capture the complexities of real-world multi-user interactions. In this study, we assess the robustness of LLMs in multi-user DST while minimizing dataset construction costs. Inspired by recent advances in LLM-based data annotation, we extend an existing DST dataset by generating utterances of a second user based on speech act theory. Our methodology systematically incorporates a second user's utterances into conversations, enabling a controlled evaluation of LLMs in multi-user settings. Experimental results reveal a significant performance drop compared to single-user DST, highlighting the limitations of current LLMs in extracting and tracking dialogue states amidst multiple speakers. Our findings emphasize the need for future research to enhance LLMs for multi-user DST scenarios, paving the way for more realistic and robust DST models.