Microsoft Dialogue Challenge: Building End-to-End Task-Completion Dialogue Systems

TL;DR

This paper presents a Dialogue Challenge aimed at advancing end-to-end task-completion dialogue systems by providing standardized datasets, an experimental platform, and evaluation methods to foster collaboration and benchmarking in the research community.

Contribution

It introduces a comprehensive benchmark with annotated datasets, simulators, and evaluation protocols for developing and assessing dialogue systems across multiple domains.

Findings

Release of annotated conversational datasets in three domains

Development of an experiment platform with built-in simulators

Evaluation framework including both simulated and human assessments

Abstract

This proposal introduces a Dialogue Challenge for building end-to-end task-completion dialogue systems, with the goal of encouraging the dialogue research community to collaborate and benchmark on standard datasets and unified experimental environment. In this special session, we will release human-annotated conversational data in three domains (movie-ticket booking, restaurant reservation, and taxi booking), as well as an experiment platform with built-in simulators in each domain, for training and evaluation purposes. The final submitted systems will be evaluated both in simulated setting and by human judges.