Multi-Level Testing of Conversational AI Systems

TL;DR

This paper proposes a multi-level testing framework for conversational AI systems, addressing the unique challenges of validating AI components and interactions at various granularities to improve reliability.

Contribution

It introduces a novel testing approach that evaluates conversational AI systems at multiple levels, from integration to multi-agent configurations, filling a gap in existing testing methods.

Findings

Developed a hierarchical testing methodology for conversational AI

Validated the approach on real-world conversational systems

Enhanced detection of integration and interaction issues

Abstract

Conversational AI systems combine AI-based solutions with the flexibility of conversational interfaces. However, most existing testing solutions do not straightforwardly adapt to the characteristics of conversational interaction or to the behavior of AI components. To address this limitation, this Ph.D. thesis investigates a new family of testing approaches for conversational AI systems, focusing on the validation of their constituent elements at different levels of granularity, from the integration between the language and the AI components, to individual conversational agents, up to multi-agent implementations of conversational AI systems