Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction

TL;DR

This paper models task-oriented dialogue consistency as a Constraint Satisfaction Problem (CSP) to detect inconsistencies and evaluate the challenges faced by large language models in maintaining dialogue coherence.

Contribution

It introduces a novel CSP-based framework for assessing dialogue consistency and demonstrates its effectiveness over traditional pipeline approaches.

Findings

CSP effectively detects dialogue inconsistencies.

State-of-the-art LLMs struggle with dialogue re-lexicalization, achieving only 0.15 accuracy.

Domain knowledge constraints are the most difficult to satisfy.

Abstract

Task-oriented dialogues must maintain consistency both within the dialogue itself, ensuring logical coherence across turns, and with the conversational domain, accurately reflecting external knowledge. We propose to conceptualize dialogue consistency as a Constraint Satisfaction Problem (CSP), wherein variables represent segments of the dialogue referencing the conversational domain, and constraints among variables reflect dialogue properties, including linguistic, conversational, and domain-based aspects. To demonstrate the feasibility of the approach, we utilize a CSP solver to detect inconsistencies in dialogues re-lexicalized by an LLM. Our findings indicate that: (i) CSP is effective to detect dialogue inconsistencies; and (ii) consistent dialogue re-lexicalization is challenging for state-of-the-art LLMs, achieving only a 0.15 accuracy rate when compared to a CSP solver. Furthermore, through an ablation study, we reveal that constraints derived from domain knowledge pose the greatest difficulty in being respected. We argue that CSP captures core properties of dialogue consistency that have been poorly considered by approaches based on component pipelines.