Increasing Faithfulness in Knowledge-Grounded Dialogue with Controllable Features

TL;DR

This paper introduces controllable features and evaluation measures to improve the faithfulness and objectivity of knowledge-grounded dialogue systems, enhancing their ability to generate evidence-based responses.

Contribution

It proposes a novel approach using additional inputs and controls during training and decoding to ensure dialogue responses are more faithful to source evidence.

Findings

Controlled models produce more objective responses

Human evaluations favor controlled models over baselines

Evaluation measures effectively disentangle response styles

Abstract

Knowledge-grounded dialogue systems are intended to convey information that is based on evidence provided in a given source text. We discuss the challenges of training a generative neural dialogue model for such systems that is controlled to stay faithful to the evidence. Existing datasets contain a mix of conversational responses that are faithful to selected evidence as well as more subjective or chit-chat style responses. We propose different evaluation measures to disentangle these different styles of responses by quantifying the informativeness and objectivity. At training time, additional inputs based on these evaluation measures are given to the dialogue model. At generation time, these additional inputs act as stylistic controls that encourage the model to generate responses that are faithful to the provided evidence. We also investigate the usage of additional controls at decoding time using resampling techniques. In addition to automatic metrics, we perform a human evaluation study where raters judge the output of these controlled generation models to be generally more objective and faithful to the evidence compared to baseline dialogue systems.