Learning Multi-Level Information for Dialogue Response Selection by Highway Recurrent Transformer

TL;DR

This paper introduces a novel Highway Recurrent Transformer model with highway attention for dialogue response selection, effectively capturing multi-level dialogue information and improving response prediction accuracy.

Contribution

It proposes a new highway attention mechanism and a recurrent transformer model tailored for dialogue response selection tasks.

Findings

Model effectively captures utterance and dialogue-level information.

Experimental results on DSTC7 demonstrate improved response selection performance.

Analysis confirms the effectiveness of the proposed modules.

Abstract

With the increasing research interest in dialogue response generation, there is an emerging branch formulating this task as selecting next sentences, where given the partial dialogue contexts, the goal is to determine the most probable next sentence. Following the recent success of the Transformer model, this paper proposes (1) a new variant of attention mechanism based on multi-head attention, called highway attention, and (2) a recurrent model based on transformer and the proposed highway attention, so-called Highway Recurrent Transformer. Experiments on the response selection task in the seventh Dialog System Technology Challenge (DSTC7) show the capability of the proposed model of modeling both utterance-level and dialogue-level information; the effectiveness of each module is further analyzed as well.