On Task-Adaptive Pretraining for Dialogue Response Selection

TL;DR

This paper investigates the effectiveness of different pretraining strategies for dialogue response selection, revealing that RoBERTa initialization and MLM+NSP tasks outperform previous methods, with NSP being particularly crucial.

Contribution

The study challenges assumptions about BERT and dialogue-specific tasks, demonstrating that RoBERTa and MLM+NSP are more effective for DRS, and introduces a new state-of-the-art on the Ubuntu dataset.

Findings

RoBERTa matches BERT in DRS performance

MLM+NSP outperforms other TAP tasks

NSP is essential for effective dialogue response selection

Abstract

Recent advancements in dialogue response selection (DRS) are based on the \textit{task-adaptive pre-training (TAP)} approach, by first initializing their model with BERT~\cite{devlin-etal-2019-bert}, and adapt to dialogue data with dialogue-specific or fine-grained pre-training tasks. However, it is uncertain whether BERT is the best initialization choice, or whether the proposed dialogue-specific fine-grained learning tasks are actually better than MLM+NSP. This paper aims to verify assumptions made in previous works and understand the source of improvements for DRS. We show that initializing with RoBERTa achieve similar performance as BERT, and MLM+NSP can outperform all previously proposed TAP tasks, during which we also contribute a new state-of-the-art on the Ubuntu corpus. Additional analyses shows that the main source of improvements comes from the TAP step, and that the NSP task is crucial to DRS, different from common NLU tasks.