Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering

TL;DR

This paper presents a transformer-based model that learns hierarchical dialogue representations through multi-task pre-training and fine-tuning, significantly improving span-based question answering performance on multiparty dialogue datasets.

Contribution

It introduces a hierarchical learning approach with novel pre-training tasks and multi-task fine-tuning for dialogue-based span QA, outperforming existing models.

Findings

Achieved 3.8% improvement over BERT on FriendsQA

Achieved 1.4% improvement over RoBERTa on FriendsQA

Demonstrated effectiveness of hierarchical dialogue modeling

Abstract

We introduce a novel approach to transformers that learns hierarchical representations in multiparty dialogue. First, three language modeling tasks are used to pre-train the transformers, token- and utterance-level language modeling and utterance order prediction, that learn both token and utterance embeddings for better understanding in dialogue contexts. Then, multi-task learning between the utterance prediction and the token span prediction is applied to fine-tune for span-based question answering (QA). Our approach is evaluated on the FriendsQA dataset and shows improvements of 3.8% and 1.4% over the two state-of-the-art transformer models, BERT and RoBERTa, respectively.