Interactive Evaluation of Dialog Track at DSTC9

TL;DR

This paper discusses the Interactive Evaluation of Dialog Track at DSTC9, focusing on developing and assessing dialog systems in interactive, real-user settings to improve open-domain response quality.

Contribution

It introduces a new interactive evaluation track with two sub-tasks, emphasizing real-user interaction and extending beyond static datasets for dialog system assessment.

Findings

Development of models capable of engaging in real-user interactions

Insights into evaluation strategies for open-domain dialog systems

Challenges faced in transitioning from static to interactive evaluation

Abstract

The ultimate goal of dialog research is to develop systems that can be effectively used in interactive settings by real users. To this end, we introduced the Interactive Evaluation of Dialog Track at the 9th Dialog System Technology Challenge. This track consisted of two sub-tasks. The first sub-task involved building knowledge-grounded response generation models. The second sub-task aimed to extend dialog models beyond static datasets by assessing them in an interactive setting with real users. Our track challenges participants to develop strong response generation models and explore strategies that extend them to back-and-forth interactions with real users. The progression from static corpora to interactive evaluation introduces unique challenges and facilitates a more thorough assessment of open-domain dialog systems. This paper provides an overview of the track, including the methodology and results. Furthermore, it provides insights into how to best evaluate open-domain dialog models