Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances

TL;DR

This paper introduces DialoFlow, a novel dialogue model that captures dynamic information flow across utterances, significantly improving response quality and providing a new automatic evaluation metric for human-bot interactions.

Contribution

The paper proposes DialoFlow with a dynamic flow mechanism and training objectives to model information dynamics, advancing beyond the flat concatenation approach in dialogue modeling.

Findings

DialoFlow outperforms DialoGPT on Reddit and DailyDialog datasets.

Flow score correlates highly with human ratings ($r=0.9$).

The approach effectively models semantic influence across dialogue turns.

Abstract

Nowadays, open-domain dialogue models can generate acceptable responses according to the historical context based on the large-scale pre-trained language models. However, they generally concatenate the dialogue history directly as the model input to predict the response, which we named as the flat pattern and ignores the dynamic information flow across dialogue utterances. In this work, we propose the DialoFlow model, in which we introduce a dynamic flow mechanism to model the context flow, and design three training objectives to capture the information dynamics across dialogue utterances by addressing the semantic influence brought about by each utterance in large-scale pre-training. Experiments on the multi-reference Reddit Dataset and DailyDialog Dataset demonstrate that our DialoFlow significantly outperforms the DialoGPT on the dialogue generation task. Besides, we propose the Flow score, an effective automatic metric for evaluating interactive human-bot conversation quality based on the pre-trained DialoFlow, which presents high chatbot-level correlation ($r=0.9$) with human ratings among 11 chatbots. Code and pre-trained models will be public. \footnote{\url{https://github.com/ictnlp/DialoFlow}}