Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation

TL;DR

This paper conducts an empirical system-wise evaluation of goal-oriented dialog systems, revealing insights into component contributions, evaluation methods, and the validity of simulated assessments versus human judgments.

Contribution

It provides a comprehensive empirical analysis of different dialog system modules, highlighting the effectiveness of fine-grained supervision and the reliability of simulated evaluations.

Findings

Fine-grained supervision improves system performance.

Single-turn component evaluation may not reflect overall system quality.

Simulated evaluation is a viable alternative to human assessment.

Abstract

There is a growing interest in developing goal-oriented dialog systems which serve users in accomplishing complex tasks through multi-turn conversations. Although many methods are devised to evaluate and improve the performance of individual dialog components, there is a lack of comprehensive empirical study on how different components contribute to the overall performance of a dialog system. In this paper, we perform a system-wise evaluation and present an empirical analysis on different types of dialog systems which are composed of different modules in different settings. Our results show that (1) a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels, (2) component-wise, single-turn evaluation results are not always consistent with the overall performance of a dialog system, and (3) despite the discrepancy between simulators and human users, simulated evaluation is still a valid alternative to the costly human evaluation especially in the early stage of development.