History-Aware Hierarchical Transformer for Multi-session Open-domain Dialogue System

TL;DR

This paper introduces HAHT, a hierarchical transformer model that effectively incorporates long-term conversation history to improve multi-session open-domain dialogue systems, outperforming baseline models in relevance and human-likeness.

Contribution

The paper presents a novel hierarchical transformer architecture that encodes and utilizes long-term conversation history for multi-session dialogue, a less explored area.

Findings

HAHT outperforms baseline models on large-scale MSC dataset.

Human evaluations show HAHT generates more human-like responses.

HAHT effectively leverages historical information for context understanding.

Abstract

With the evolution of pre-trained language models, current open-domain dialogue systems have achieved great progress in conducting one-session conversations. In contrast, Multi-Session Conversation (MSC), which consists of multiple sessions over a long term with the same user, is under-investigated. In this paper, we propose History-Aware Hierarchical Transformer (HAHT) for multi-session open-domain dialogue. HAHT maintains a long-term memory of history conversations and utilizes history information to understand current conversation context and generate well-informed and context-relevant responses. Specifically, HAHT first encodes history conversation sessions hierarchically into a history memory. Then, HAHT leverages historical information to facilitate the understanding of the current conversation context by encoding the history memory together with the current context with attention-based mechanisms. Finally, to explicitly utilize historical information, HAHT uses a history-aware response generator that switches between a generic vocabulary and a history-aware vocabulary. Experimental results on a large-scale MSC dataset suggest that the proposed HAHT model consistently outperforms baseline models. Human evaluation results support that HAHT generates more human-like, context-relevant and history-relevant responses than baseline models.