Learning Locality and Isotropy in Dialogue Modeling

TL;DR

This paper introduces SimDRC, a method to improve dialogue representations by enhancing locality and isotropy, leading to better performance on multiple dialogue tasks.

Contribution

The paper proposes a simple calibration method, SimDRC, to build isotropic and conversational feature spaces, addressing anisotropy and loss of conversation structure in dialogue models.

Findings

Significant performance improvements on three dialogue tasks.

Outperforms current state-of-the-art models.

Validated effectiveness through automatic and human evaluations.

Abstract

Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models. However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy. In this paper, we find that the generated representations are also not conversational, losing the conversation structure information during the context modeling stage. To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces. Experimental results show that our approach significantly outperforms the current state-of-the-art models on three dialogue tasks across the automatic and human evaluation metrics. More in-depth analyses further confirm the effectiveness of our proposed approach.