Context-Aware Language Modeling for Goal-Oriented Dialogue Systems

TL;DR

This paper introduces CALM, a novel method for fine-tuning language models to improve goal-oriented dialogue systems by balancing language quality and task success, demonstrated on a flight-booking task.

Contribution

It formulates goal-oriented dialogue as a partially observed Markov decision process and applies control techniques to enhance task performance of language models.

Findings

CALM outperforms previous methods by 7% in task success.

The approach achieves human-level performance on the flight-booking task.

Training strategies improve the model's focus on task-specific goals.

Abstract

Goal-oriented dialogue systems face a trade-off between fluent language generation and task-specific control. While supervised learning with large language models is capable of producing realistic text, how to steer such responses towards completing a specific task without sacrificing language quality remains an open question. In this work, we formulate goal-oriented dialogue as a partially observed Markov decision process, interpreting the language model as a representation of both the dynamics and the policy. This view allows us to extend techniques from learning-based control, such as task relabeling, to derive a simple and effective method to finetune language models in a goal-aware way, leading to significantly improved task performance. We additionally introduce a number of training strategies that serve to better focus the model on the task at hand. We evaluate our method, Context-Aware Language Models (CALM), on a practical flight-booking task using AirDialogue. Empirically, CALM outperforms the state-of-the-art method by 7% in terms of task success, matching human-level task performance.