Coherent Dialogue with Attention-based Language Models

TL;DR

This paper introduces an attention-based RNN dialogue model that dynamically adjusts its focus on conversation history, significantly improving coherence and relevance in open and closed-domain dialogues, outperforming existing models.

Contribution

The paper proposes a novel dynamic attention mechanism for RNN dialogue models, enhancing their ability to maintain coherence over long conversations and outperforming more complex memory models.

Findings

Significant improvements over state-of-the-art on dialogue coherence metrics

Dynamic attention enables better long-distance memory than LSTM/GRU

Topic reranking further enhances dialogue coherence

Abstract

We model coherent conversation continuation via RNN-based dialogue models equipped with a dynamic attention mechanism. Our attention-RNN language model dynamically increases the scope of attention on the history as the conversation continues, as opposed to standard attention (or alignment) models with a fixed input scope in a sequence-to-sequence model. This allows each generated word to be associated with the most relevant words in its corresponding conversation history. We evaluate the model on two popular dialogue datasets, the open-domain MovieTriples dataset and the closed-domain Ubuntu Troubleshoot dataset, and achieve significant improvements over the state-of-the-art and baselines on several metrics, including complementary diversity-based metrics, human evaluation, and qualitative visualizations. We also show that a vanilla RNN with dynamic attention outperforms more complex memory models (e.g., LSTM and GRU) by allowing for flexible, long-distance memory. We promote further coherence via topic modeling-based reranking.