Similarity Based Label Smoothing For Dialogue Generation

TL;DR

This paper proposes a data-dependent, similarity-based label smoothing technique for dialogue generation, replacing the uniform distribution of incorrect targets with a semantic similarity-informed distribution, leading to improved performance.

Contribution

It introduces a novel similarity-based weighting method for label smoothing in dialogue systems, enhancing training by incorporating semantic relations among words.

Findings

Significant performance improvements over standard label smoothing.

Effective incorporation of semantic similarity into label smoothing.

Validated on two open domain dialogue datasets.

Abstract

Generative neural conversational systems are generally trained with the objective of minimizing the entropy loss between the training "hard" targets and the predicted logits. Often, performance gains and improved generalization can be achieved by using regularization techniques like label smoothing, which converts the training "hard" targets to "soft" targets. However, label smoothing enforces a data independent uniform distribution on the incorrect training targets, which leads to an incorrect assumption of equi-probable incorrect targets for each correct target. In this paper we propose and experiment with incorporating data dependent word similarity based weighing methods to transforms the uniform distribution of the incorrect target probabilities in label smoothing, to a more natural distribution based on semantics. We introduce hyperparameters to control the incorrect target distribution, and report significant performance gains over networks trained using standard label smoothing based loss, on two standard open domain dialogue corpora.