Toward Interpretability of Dual-Encoder Models for Dialogue Response Suggestions

TL;DR

This paper introduces an attentive dual encoder model with a novel regularization loss and residual layer to enhance interpretability and accuracy in dialogue response suggestion tasks.

Contribution

It proposes a new attentive dual encoder architecture with mutual information regularization and residual connections for improved interpretability and performance.

Findings

Improved Recall@1 accuracy on Persona and Ubuntu datasets.

Enhanced interpretability through visualization of important words.

Model outperforms existing methods in response suggestion tasks.

Abstract

This work shows how to improve and interpret the commonly used dual encoder model for response suggestion in dialogue. We present an attentive dual encoder model that includes an attention mechanism on top of the extracted word-level features from two encoders, one for context and one for label respectively. To improve the interpretability in the dual encoder models, we design a novel regularization loss to minimize the mutual information between unimportant words and desired labels, in addition to the original attention method, so that important words are emphasized while unimportant words are de-emphasized. This can help not only with model interpretability, but can also further improve model accuracy. We propose an approximation method that uses a neural network to calculate the mutual information. Furthermore, by adding a residual layer between raw word embeddings and the final encoded context feature, word-level interpretability is preserved at the final prediction of the model. We compare the proposed model with existing methods for the dialogue response task on two public datasets (Persona and Ubuntu). The experiments demonstrate the effectiveness of the proposed model in terms of better Recall@1 accuracy and visualized interpretability.