Pragmatically Appropriate Diversity for Dialogue Evaluation

TL;DR

This paper introduces Pragmatically Appropriate Diversity, a new concept linking speech acts to response diversity in dialogue, supported by human studies and proposing improved evaluation methods for dialogue systems.

Contribution

It defines Pragmatically Appropriate Diversity, demonstrates its relevance through human-created datasets, and proposes a novel human evaluation task for dialogue response diversity.

Findings

Speech acts signal the potential for response diversity

Human judgments align with Pragmatically Appropriate Diversity

Diversity expectations should vary by speech act

Abstract

Linguistic pragmatics state that a conversation's underlying speech acts can constrain the type of response which is appropriate at each turn in the conversation. When generating dialogue responses, neural dialogue agents struggle to produce diverse responses. Currently, dialogue diversity is assessed using automatic metrics, but the underlying speech acts do not inform these metrics. To remedy this, we propose the notion of Pragmatically Appropriate Diversity, defined as the extent to which a conversation creates and constrains the creation of multiple diverse responses. Using a human-created multi-response dataset, we find significant support for the hypothesis that speech acts provide a signal for the diversity of the set of next responses. Building on this result, we propose a new human evaluation task where creative writers predict the extent to which conversations inspire the creation of multiple diverse responses. Our studies find that writers' judgments align with the Pragmatically Appropriate Diversity of conversations. Our work suggests that expectations for diversity metric scores should vary depending on the speech act.