MORTAR: Multi-turn Metamorphic Testing for LLM-based Dialogue Systems

TL;DR

MORTAR introduces an automated multi-turn metamorphic testing approach for LLM-based dialogue systems, effectively revealing more diverse and higher-quality bugs compared to single-turn methods, addressing the challenge of the multi-turn oracle problem.

Contribution

It formalizes multi-turn dialogue testing with automated question-answer dialogue generation and metamorphic relations, improving bug detection effectiveness without LLM judges.

Findings

Revealed over 150% more bugs than single-turn baseline.

Detected higher-quality bugs in diversity, precision, and uniqueness.

Effective in testing six popular LLM-based dialogue systems.

Abstract

With the widespread application of LLM-based dialogue systems in daily life, quality assurance has become more important than ever. Recent research has successfully introduced methods to identify unexpected behaviour in single-turn testing scenarios. However, multi-turn interaction is the common real-world usage of dialogue systems, yet testing methods for such interactions remain underexplored. This is largely due to the oracle problem in multi-turn testing, which continues to pose a significant challenge for dialogue system developers and researchers. In this paper, we propose MORTAR, a metamorphic multi-turn dialogue testing approach, which mitigates the test oracle problem in testing LLM-based dialogue systems. MORTAR formalises the multi-turn testing for dialogue systems, and automates the generation of question-answer dialogue test cases with multiple dialogue-level perturbations and metamorphic relations (MRs). The automated MR matching mechanism allows MORTAR more flexibility and efficiency in metamorphic testing. The proposed approach is fully automated without reliance on LLM judges. In testing six popular LLM-based dialogue systems, MORTAR reaches significantly better effectiveness with over 150\% more bugs revealed per test case when compared to the single-turn metamorphic testing baseline. Regarding the quality of bugs, MORTAR reveals higher-quality bugs in terms of diversity, precision and uniqueness. MORTAR is expected to inspire more multi-turn testing approaches, and assist developers in evaluating the dialogue system performance more comprehensively with constrained test resources and budget.