A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models

TL;DR

This paper introduces a methodology to identify diverse evaluation items for practical dialogue systems by leveraging business-dialogue system alignment models, aiming to enhance system development and research.

Contribution

It proposes a novel methodology based on business-IT alignment models to systematically identify evaluation items for dialogue systems, filling a gap in existing evaluation frameworks.

Findings

A generic model for constructing business-dialogue system alignment models.

Identification of evaluation items that extend beyond traditional user satisfaction metrics.

Potential to guide new research directions in dialogue system evaluation.

Abstract

This paper proposes a methodology for identifying evaluation items for practical dialogue systems. Traditionally, user satisfaction and user experiences have been the primary metrics for evaluating dialogue systems. However, there are various other evaluation items to consider when developing and operating practical dialogue systems, and such evaluation items are expected to lead to new research topics. So far, there has been no methodology for identifying these evaluation items. We propose identifying evaluation items based on business-dialogue system alignment models, which are applications of business-IT alignment models used in the development and operation of practical IT systems. We also present a generic model that facilitates the construction of a business-dialogue system alignment model for each dialogue system.