An Application of Reinforcement Learning to Dialogue Strategy Selection in a Spoken Dialogue System for Email

TL;DR

This paper presents a reinforcement learning approach combined with performance modeling to enable a spoken dialogue system to learn optimal strategies from user interactions, improving email access over the phone.

Contribution

It introduces a novel method integrating Q-learning and the PARADISE framework for dialogue strategy optimization in spoken systems.

Findings

ELVIS learned to optimize initiative, message reading, and email summarization strategies.

The system improved performance based on user interactions and dialogue success.

Experimental results demonstrated effective strategy adaptation.

Abstract

This paper describes a novel method by which a spoken dialogue system can learn to choose an optimal dialogue strategy from its experience interacting with human users. The method is based on a combination of reinforcement learning and performance modeling of spoken dialogue systems. The reinforcement learning component applies Q-learning (Watkins, 1989), while the performance modeling component applies the PARADISE evaluation framework (Walker et al., 1997) to learn the performance function (reward) used in reinforcement learning. We illustrate the method with a spoken dialogue system named ELVIS (EmaiL Voice Interactive System), that supports access to email over the phone. We conduct a set of experiments for training an optimal dialogue strategy on a corpus of 219 dialogues in which human users interact with ELVIS over the phone. We then test that strategy on a corpus of 18 dialogues. We show that ELVIS can learn to optimize its strategy selection for agent initiative, for reading messages, and for summarizing email folders.