UniPCM: Universal Pre-trained Conversation Model with Task-aware Automatic Prompt

TL;DR

UniPCM is a versatile pre-trained conversation model that uses automatic prompt generation to enhance multi-task learning, achieving state-of-the-art results across diverse dialog tasks with improved robustness and transferability.

Contribution

This work introduces TAP for automatic prompt generation, enabling scalable multi-task pre-training and superior performance in dialog systems without relying on human-crafted prompts.

Findings

Achieves SOTA results on 9 datasets across dialog tasks

Demonstrates robustness to prompt variations and low-resource scenarios

TAP-generated prompts match crowdsourced quality

Abstract

Recent research has shown that multi-task pre-training greatly improves the model's robustness and transfer ability, which is crucial for building a high-quality dialog system. However, most previous works on multi-task pre-training rely heavily on human-defined input format or prompt, which is not optimal in quality and quantity. In this work, we propose to use Task-based Automatic Prompt generation (TAP) to automatically generate high-quality prompts. Using the high-quality prompts generated, we scale the corpus of the pre-trained conversation model to 122 datasets from 15 dialog-related tasks, resulting in Universal Pre-trained Conversation Model (UniPCM), a powerful foundation model for various conversational tasks and different dialog systems. Extensive experiments have shown that UniPCM is robust to input prompts and capable of various dialog-related tasks. Moreover, UniPCM has strong transfer ability and excels at low resource scenarios, achieving SOTA results on 9 different datasets ranging from task-oriented dialog to open-domain conversation. Furthermore, we are amazed to find that TAP can generate prompts on par with those collected with crowdsourcing. The code is released with the paper.