# Hello, It's GPT-2 -- How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems

## TL;DR

This paper explores using pretrained language models to develop task-oriented dialogue systems that require less task-specific data, leveraging transfer learning to improve multi-domain dialogue performance.

## Contribution

It introduces a text-only, pretraining-based dialogue model that bypasses explicit modules, demonstrating competitive results on multi-domain datasets.

## Key findings

- Model performs on par with strong task-specific baselines.
- Pretraining helps mitigate data scarcity in dialogue systems.
- Approach supports more engaging and eloquent conversations.

## Abstract

Data scarcity is a long-standing and crucial challenge that hinders quick development of task-oriented dialogue systems across multiple domains: task-oriented dialogue models are expected to learn grammar, syntax, dialogue reasoning, decision making, and language generation from absurdly small amounts of task-specific data. In this paper, we demonstrate that recent progress in language modeling pre-training and transfer learning shows promise to overcome this problem. We propose a task-oriented dialogue model that operates solely on text input: it effectively bypasses explicit policy and language generation modules. Building on top of the TransferTransfo framework (Wolf et al., 2019) and generative model pre-training (Radford et al., 2019), we validate the approach on complex multi-domain task-oriented dialogues from the MultiWOZ dataset. Our automatic and human evaluations show that the proposed model is on par with a strong task-specific neural baseline. In the long run, our approach holds promise to mitigate the data scarcity problem, and to support the construction of more engaging and more eloquent task-oriented conversational agents.