Bidirectional Empowerment of Metamorphic Testing and Large Language Models: A Systematic Survey

TL;DR

This systematic survey explores the reciprocal enhancement between metamorphic testing and large language models, highlighting their combined potential to improve AI testing and validation methods.

Contribution

It introduces a taxonomy of how metamorphic testing and LLMs mutually empower each other, covering verification, relation discovery, and automated testing techniques.

Findings

Reviewed 93 primary studies on MT and LLMs

Proposed a taxonomy of bidirectional empowerment

Identified future directions for AI quality assurance

Abstract

Large language models (LLMs) have introduced substantial challenges to software quality assurance due to their generative, probabilistic, and open-ended nature, which intensifies the oracle problem and limits the applicability of traditional testing methods. Metamorphic testing (MT), which checks necessary relations among multiple related executions rather than relying on exact expected outputs, has emerged as a promising approach for testing LLMs and other oracle-deficient systems. At the same time, the strong semantic understanding, reasoning, and code generation capabilities of LLMs create new opportunities to automate the traditionally labor-intensive phases of MT. This survey systematically reviews 93 primary studies and characterizes this reciprocal relationship as the bidirectional empowerment of MT and LLMs. We propose a taxonomy spanning two complementary directions: MT for LLMs, which uses MT to verify, validate, assess, and understand LLMs and LLM-based systems across issues such as hallucination, fairness, robustness, code reliability, retrieval-augmented generation, dialogue, and autonomous agents; and LLMs for MT, which leverages LLMs to support metamorphic relation discovery, input transformation and synthesis, executable test implementation, and agentic closed-loop testing. By synthesizing these developments, this survey provides a structured foundation for understanding the evolving synergy between MT and LLMs and highlights future directions for building more rigorous, scalable, and trustworthy AI quality assurance methodologies.