What makes a good conversation? How controllable attributes affect human judgments

TL;DR

This paper investigates how controllable neural text generation methods can influence key attributes of dialogue quality, such as repetition and specificity, through large-scale human evaluations on multi-turn conversations.

Contribution

It introduces methods to control four dialogue attributes and demonstrates their impact on human judgments, improving conversation quality in neural dialogue systems.

Findings

Controlling attributes like repetition and specificity improves human-rated conversation quality.

Large-scale human evaluation confirms the effectiveness of controllable methods.

Combining attribute controls yields better overall dialogue quality.

Abstract

A good conversation requires balance -- between simplicity and detail; staying on topic and changing it; asking questions and answering them. Although dialogue agents are commonly evaluated via human judgments of overall quality, the relationship between quality and these individual factors is less well-studied. In this work, we examine two controllable neural text generation methods, conditional training and weighted decoding, in order to control four important attributes for chitchat dialogue: repetition, specificity, response-relatedness and question-asking. We conduct a large-scale human evaluation to measure the effect of these control parameters on multi-turn interactive conversations on the PersonaChat task. We provide a detailed analysis of their relationship to high-level aspects of conversation, and show that by controlling combinations of these variables our models obtain clear improvements in human quality judgments.