Achieving Fluency and Coherency in Task-oriented Dialog

TL;DR

This paper presents a hybrid approach combining nearest neighbor and Seq2Seq models to improve fluency, coherence, and accuracy of external actions in task-oriented dialog systems, especially for customer support.

Contribution

It introduces a hybrid model that leverages neural embeddings and nearest neighbor techniques to enhance response fluency and external action accuracy in real-world dialog settings.

Findings

78% improvement in fluency scores

130% improvement in external action accuracy

Effective for script-driven customer support dialogs

Abstract

We consider real world task-oriented dialog settings, where agents need to generate both fluent natural language responses and correct external actions like database queries and updates. We demonstrate that, when applied to customer support chat transcripts, Sequence to Sequence (Seq2Seq) models often generate short, incoherent and ungrammatical natural language responses that are dominated by words that occur with high frequency in the training data. These phenomena do not arise in synthetic datasets such as bAbI, where we show Seq2Seq models are nearly perfect. We develop techniques to learn embeddings that succinctly capture relevant information from the dialog history, and demonstrate that nearest neighbor based approaches in this learned neural embedding space generate more fluent responses. However, we see that these methods are not able to accurately predict when to execute an external action. We show how to combine nearest neighbor and Seq2Seq methods in a hybrid model, where nearest neighbor is used to generate fluent responses and Seq2Seq type models ensure dialog coherency and generate accurate external actions. We show that this approach is well suited for customer support scenarios, where agents' responses are typically script-driven, and correct external actions are critically important. The hybrid model on the customer support data achieves a 78% relative improvement in fluency scores, and a 130% improvement in accuracy of external calls.