On Task-Level Dialogue Composition of Generative Transformer Model

TL;DR

This paper investigates the challenge of enabling generative Transformer models to compose multiple tasks within a single dialogue, proposing data augmentation and invariant representation techniques to improve multi-task dialogue generation.

Contribution

It introduces methods to train Transformer models for multi-task dialogue composition using synthetic data and auxiliary loss to enhance representation invariance.

Findings

Transformers struggle to learn multi-task composition from single-task dialogues.

Synthetic multi-task dialogue data can aid training.

Forcing encoder invariance improves multi-task dialogue modeling.

Abstract

Task-oriented dialogue systems help users accomplish tasks such as booking a movie ticket and ordering food via conversation. Generative models parameterized by a deep neural network are widely used for next turn response generation in such systems. It is natural for users of the system to want to accomplish multiple tasks within the same conversation, but the ability of generative models to compose multiple tasks is not well studied. In this work, we begin by studying the effect of training human-human task-oriented dialogues towards improving the ability to compose multiple tasks on Transformer generative models. To that end, we propose and explore two solutions: (1) creating synthetic multiple task dialogue data for training from human-human single task dialogue and (2) forcing the encoder representation to be invariant to single and multiple task dialogues using an auxiliary loss. The results from our experiments highlight the difficulty of even the sophisticated variant of transformer model in learning to compose multiple tasks from single task dialogues.