Evaluating Competing Agent Strategies for a Voice Email Agent

TL;DR

This study compares system-initiative and mixed-initiative dialog strategies in a voice email agent, evaluating their performance and user interaction effectiveness using the PARADISE framework.

Contribution

It provides an experimental comparison of dialog strategies in voice agents and demonstrates the application of the PARADISE evaluation framework.

Findings

Mixed-initiative strategy improves user satisfaction over system-initiative.

Performance varies across tasks and strategies, highlighting context-dependent effectiveness.

The PARADISE framework effectively evaluates dialog system performance.

Abstract

This paper reports experimental results comparing a mixed-initiative to a system-initiative dialog strategy in the context of a personal voice email agent. To independently test the effects of dialog strategy and user expertise, users interact with either the system-initiative or the mixed-initiative agent to perform three successive tasks which are identical for both agents. We report performance comparisons across agent strategies as well as over tasks. This evaluation utilizes and tests the PARADISE evaluation framework, and discusses the performance function derivable from the experimental data.