Unsupervised End-to-End Task-Oriented Dialogue with LLMs: The Power of the Noisy Channel

TL;DR

This paper introduces an unsupervised method for building task-oriented dialogue systems using large language models, leveraging only API schemas and unlabeled dialogues, significantly improving success rates over baseline models.

Contribution

It presents a novel unsupervised approach using expectation-maximization and noisy channel models to infer dialogue annotations without labeled data.

Findings

More than doubles the dialogue success rate compared to GPT-3.5 baseline.

Effective use of unlabeled dialogues and API schemas for dialogue system training.

Demonstrates feasibility of fully unsupervised task-oriented dialogue systems.

Abstract

Training task-oriented dialogue systems typically requires turn-level annotations for interacting with their APIs: e.g. a dialogue state and the system actions taken at each step. These annotations can be costly to produce, error-prone, and require both domain and annotation expertise. With advances in LLMs, we hypothesize that unlabeled data and a schema definition are sufficient for building a working task-oriented dialogue system, completely unsupervised. We consider a novel unsupervised setting of only (1) a well-defined API schema (2) a set of unlabeled dialogues between a user and agent. We propose an innovative approach using expectation-maximization (EM) that infers turn-level annotations as latent variables using a noisy channel model to build an end-to-end dialogue agent. Evaluating our approach on the MultiWOZ benchmark, our method more than doubles the dialogue success rate of a strong GPT-3.5 baseline.