OPAL: Ontology-Aware Pretrained Language Model for End-to-End Task-Oriented Dialogue

TL;DR

OPAL is an ontology-aware pretrained language model designed for end-to-end task-oriented dialogue, utilizing a two-phase pretraining approach to improve performance even with limited dialogue data.

Contribution

The paper introduces a novel two-phase pretraining method that incorporates structured ontology information, enhancing task-oriented dialogue models without requiring large annotated datasets.

Findings

Achieves competitive performance on CamRest676 and MultiWOZ benchmarks.

Effectively simulates dialogue state tracking and response generation tasks.

Boosts performance even without using task-specific dialogue data.

Abstract

This paper presents an ontology-aware pretrained language model (OPAL) for end-to-end task-oriented dialogue (TOD). Unlike chit-chat dialogue models, task-oriented dialogue models fulfill at least two task-specific modules: dialogue state tracker (DST) and response generator (RG). The dialogue state consists of the domain-slot-value triples, which are regarded as the user's constraints to search the domain-related databases. The large-scale task-oriented dialogue data with the annotated structured dialogue state usually are inaccessible. It prevents the development of the pretrained language model for the task-oriented dialogue. We propose a simple yet effective pretraining method to alleviate this problem, which consists of two pretraining phases. The first phase is to pretrain on large-scale contextual text data, where the structured information of the text is extracted by the information extracting tool. To bridge the gap between the pretraining method and downstream tasks, we design two pretraining tasks: ontology-like triple recovery and next-text generation, which simulates the DST and RG, respectively. The second phase is to fine-tune the pretrained model on the TOD data. The experimental results show that our proposed method achieves an exciting boost and get competitive performance even without any TOD data on CamRest676 and MultiWOZ benchmarks.