Don't Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training

TL;DR

This paper introduces an extension of unlikelihood training to address common issues in dialogue generation, such as repetition, copying, overuse of frequent words, and logical flaws, leading to more consistent and human-like responses.

Contribution

It extends unlikelihood training to mitigate multiple dialogue generation problems and applies it to improve logical consistency and overall quality.

Findings

Reduced repetition and copying in generated dialogues

Improved logical consistency in responses

Enhanced alignment with human-like distributions

Abstract

Generative dialogue models currently suffer from a number of problems which standard maximum likelihood training does not address. They tend to produce generations that (i) rely too much on copying from the context, (ii) contain repetitions within utterances, (iii) overuse frequent words, and (iv) at a deeper level, contain logical flaws. In this work we show how all of these problems can be addressed by extending the recently introduced unlikelihood loss (Welleck et al., 2019) to these cases. We show that appropriate loss functions which regularize generated outputs to match human distributions are effective for the first three issues. For the last important general issue, we show applying unlikelihood to collected data of what a model should not do is effective for improving logical consistency, potentially paving the way to generative models with greater reasoning ability. We demonstrate the efficacy of our approach across several dialogue tasks.