Reducing Sensitivity on Speaker Names for Text Generation from Dialogues

TL;DR

This paper investigates how pre-trained dialogue models are sensitive to speaker name changes, proposes methods to reduce this sensitivity, and benchmarks their effectiveness to improve fairness and robustness in dialogue generation.

Contribution

It introduces a quantitative measure for speaker name sensitivity, evaluates existing methods, and proposes a novel approach, providing a comprehensive benchmark for this problem.

Findings

Our approach significantly reduces speaker name sensitivity.

Models maintain high quality of dialogue generation after applying our method.

Extensive experiments validate the effectiveness of the proposed solutions.

Abstract

Changing speaker names consistently throughout a dialogue should not affect its meaning and corresponding outputs for text generation from dialogues. However, pre-trained language models, serving as the backbone for dialogue-processing tasks, have shown to be sensitive to nuances. This may result in unfairness in real-world applications. No comprehensive analysis of this problem has been done in the past. In this work, we propose to quantitatively measure a model's sensitivity on speaker names, and comprehensively evaluate a number of known methods for reducing speaker name sensitivity, including a novel approach of our own. Extensive experiments on multiple datasets provide a benchmark for this problem and show the favorable performance of our approach in sensitivity reduction and quality of generation.