A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding

TL;DR

This paper introduces a zero-shot, open-vocabulary dialogue state tracking system that reformulates the task as question-answering, achieving significant accuracy improvements and reduced API calls without relying on predefined ontologies.

Contribution

The paper presents a novel zero-shot DST pipeline that reformulates the task as question-answering and employs self-refining prompts, enabling dynamic slot handling and improved performance.

Findings

Up to 20% better Joint Goal Accuracy over SOTA methods.

Achieves 90% fewer API requests compared to existing LLM-based systems.

Effectively adapts to new slots without fixed ontologies.

Abstract

Dialogue State Tracking (DST) is crucial for understanding user needs and executing appropriate system actions in task-oriented dialogues. Majority of existing DST methods are designed to work within predefined ontologies and assume the availability of gold domain labels, struggling with adapting to new slots values. While Large Language Models (LLMs)-based systems show promising zero-shot DST performance, they either require extensive computational resources or they underperform existing fully-trained systems, limiting their practicality. To address these limitations, we propose a zero-shot, open-vocabulary system that integrates domain classification and DST in a single pipeline. Our approach includes reformulating DST as a question-answering task for less capable models and employing self-refining prompts for more adaptable ones. Our system does not rely on fixed slot values defined in the ontology allowing the system to adapt dynamically. We compare our approach with existing SOTA, and show that it provides up to 20% better Joint Goal Accuracy (JGA) over previous methods on datasets like Multi-WOZ 2.1, with up to 90% fewer requests to the LLM API.