Measuring and Improving Semantic Diversity of Dialogue Generation

TL;DR

This paper introduces a new automatic metric for measuring semantic diversity in dialogue responses and proposes a learning method to enhance this diversity, outperforming existing approaches in both automatic and human evaluations.

Contribution

The paper presents a novel semantic diversity metric for dialogue responses and a simple learning method that improves response diversity and coherence.

Findings

The new metric aligns better with human judgments than lexical diversity metrics.

The proposed learning method enhances semantic diversity and response coherence.

Automatic and human evaluations confirm the effectiveness of the approach.

Abstract

Response diversity has become an important criterion for evaluating the quality of open-domain dialogue generation models. However, current evaluation metrics for response diversity often fail to capture the semantic diversity of generated responses, as they mainly consider lexical aspects of the generated responses. In this paper, we introduce a new automatic evaluation metric to measure the semantic diversity of generated responses. Through human evaluation, we demonstrate that our proposed metric captures human judgments on response diversity better than existing lexical-level diversity metrics. Furthermore, motivated by analyzing an existing dialogue dataset, we propose a simple yet effective learning method that improves the semantic diversity of generated responses. Our learning method weights training samples based on the semantic distribution of the training set. We show that our learning method improves response diversity and coherency better than other baseline methods through automatic and human evaluation.