An Automated Testing Framework for Conversational Agents

TL;DR

This paper presents a work-in-progress automated testing framework for conversational agents, aiming to improve testing efficiency and quality, while addressing challenges like specifying expected behaviors and semantic utterance comparison.

Contribution

It introduces a new automated testing framework for conversational agents implemented in Python, highlighting key research challenges in test oracle specification and semantic comparison.

Findings

Framework facilitates writing, executing, and maintaining test cases.

Addresses key research problems in test oracle specification.

Highlights challenges in semantic comparison of utterances.

Abstract

Conversational agents are systems with a conversational interface that afford interaction in spoken language. These systems are becoming prevalent and are preferred in various contexts and for many users. Despite their increasing success, the automated testing infrastructure to support the effective and efficient development of such systems compared to traditional software systems is still limited. Automated testing framework for conversational systems can improve the quality of these systems by assisting developers to write, execute, and maintain test cases. In this paper, we introduce our work-in-progress automated testing framework, and its realization in the Python programming language. We discuss some research problems in the development of such an automated testing framework for conversational agents. In particular, we point out the problems of the specification of the expected behavior, known as test oracles, and semantic comparison of utterances.