TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue

TL;DR

TOD-BERT is a specialized pre-trained language model designed for task-oriented dialogue, incorporating dialogue-specific features and objectives to improve performance across multiple dialogue understanding tasks.

Contribution

The paper introduces TOD-BERT, a pre-trained model that unifies dialogue datasets and employs novel training objectives to enhance task-oriented dialogue understanding.

Findings

Outperforms BERT on four dialogue tasks

Shows strong few-shot learning capabilities

Effectively models dialogue-specific language patterns

Abstract

The underlying difference of linguistic patterns between general text and task-oriented dialogue makes existing pre-trained language models less useful in practice. In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling. We propose a contrastive objective function to simulate the response selection task. Our pre-trained task-oriented dialogue BERT (TOD-BERT) outperforms strong baselines like BERT on four downstream task-oriented dialogue applications, including intention recognition, dialogue state tracking, dialogue act prediction, and response selection. We also show that TOD-BERT has a stronger few-shot ability that can mitigate the data scarcity problem for task-oriented dialogue.