Counterfactual Off-Policy Training for Neural Response Generation

TL;DR

This paper introduces a counterfactual reasoning approach for neural response generation in dialogue systems, improving response quality by exploring alternative responses and training with adversarial learning.

Contribution

It proposes a novel counterfactual reasoning model for dialogue generation that infers and synthesizes higher-quality responses, enhancing exploration of potential response space.

Findings

Significantly outperforms HRED model

Outperforms conventional adversarial learning methods

Effective on DailyDialog dataset

Abstract

Open-domain dialogue generation suffers from the data insufficiency problem due to the vast size of potential responses. In this paper, we propose to explore potential responses by counterfactual reasoning. Given an observed response, the counterfactual reasoning model automatically infers the outcome of an alternative policy that could have been taken. The resulting counterfactual response synthesized in hindsight is of higher quality than the response synthesized from scratch. Training on the counterfactual responses under the adversarial learning framework helps to explore the high-reward area of the potential response space. An empirical study on the DailyDialog dataset shows that our approach significantly outperforms the HRED model as well as the conventional adversarial learning approaches.