A Task-oriented Dialog Model with Task-progressive and Policy-aware Pre-training

TL;DR

This paper introduces a task-progressive, policy-aware pre-training approach for task-oriented dialog models, improving performance on benchmarks with fewer parameters and less data.

Contribution

It proposes a novel multi-stage pre-training framework with policy-aware tasks to better capture dialog dynamics and policy information.

Findings

Outperforms previous state-of-the-art models on MultiWOZ and In-Car benchmarks.

Uses only 18% of parameters and 25% of pre-training data compared to GALAXY.

Achieves superior results with a task-progressive and policy-aware pre-training strategy.

Abstract

Pre-trained conversation models (PCMs) have achieved promising progress in recent years. However, existing PCMs for Task-oriented dialog (TOD) are insufficient for capturing the sequential nature of the TOD-related tasks, as well as for learning dialog policy information. To alleviate these problems, this paper proposes a task-progressive PCM with two policy-aware pre-training tasks. The model is pre-trained through three stages where TOD-related tasks are progressively employed according to the task logic of the TOD system. A global policy consistency task is designed to capture the multi-turn dialog policy sequential relation, and an act-based contrastive learning task is designed to capture similarities among samples with the same dialog policy. Our model achieves better results on both MultiWOZ and In-Car end-to-end dialog modeling benchmarks with only 18\% parameters and 25\% pre-training data compared to the previous state-of-the-art PCM, GALAXY.