Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System

TL;DR

PPTOD is a unified pre-trained model that improves task-oriented dialogue systems by learning from diverse dialogue data, achieving state-of-the-art results across multiple benchmarks with more accurate and coherent responses.

Contribution

The paper introduces PPTOD, a novel plug-and-play multi-task pre-training strategy for TOD systems that reduces error propagation and annotation overhead.

Findings

Achieves new state-of-the-art on three TOD benchmarks.

Performs well in both high-resource and low-resource settings.

Generated responses are more factually correct and semantically coherent.

Abstract

Pre-trained language models have been recently shown to benefit task-oriented dialogue (TOD) systems. Despite their success, existing methods often formulate this task as a cascaded generation problem which can lead to error accumulation across different sub-tasks and greater data annotation overhead. In this study, we present PPTOD, a unified plug-and-play model for task-oriented dialogue. In addition, we introduce a new dialogue multi-task pre-training strategy that allows the model to learn the primary TOD task completion skills from heterogeneous dialog corpora. We extensively test our model on three benchmark TOD tasks, including end-to-end dialogue modelling, dialogue state tracking, and intent classification. Experimental results show that PPTOD achieves new state of the art on all evaluated tasks in both high-resource and low-resource scenarios. Furthermore, comparisons against previous SOTA methods show that the responses generated by PPTOD are more factually correct and semantically coherent as judged by human annotators.