Learning from Perturbations: Diverse and Informative Dialogue Generation with Inverse Adversarial Training

TL;DR

This paper introduces Inverse Adversarial Training (IAT), a novel method for training neural dialogue systems that enhances response diversity and dialogue history modeling by leveraging perturbations and inverse rewards.

Contribution

The paper proposes IAT, a new adversarial training algorithm that improves dialogue response diversity and history modeling by focusing on response sensitivity to history perturbations.

Findings

IAT produces more diverse responses on benchmark datasets.

IAT better captures dialogue history compared to standard methods.

Identifies limitations of MMI-based diversity methods.

Abstract

In this paper, we propose Inverse Adversarial Training (IAT) algorithm for training neural dialogue systems to avoid generic responses and model dialogue history better. In contrast to standard adversarial training algorithms, IAT encourages the model to be sensitive to the perturbation in the dialogue history and therefore learning from perturbations. By giving higher rewards for responses whose output probability reduces more significantly when dialogue history is perturbed, the model is encouraged to generate more diverse and consistent responses. By penalizing the model when generating the same response given perturbed dialogue history, the model is forced to better capture dialogue history and generate more informative responses. Experimental results on two benchmark datasets show that our approach can better model dialogue history and generate more diverse and consistent responses. In addition, we point out a problem of the widely used maximum mutual information (MMI) based methods for improving the diversity of dialogue response generation models and demonstrate it empirically.