Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol

TL;DR

This paper analyzes the unique challenges of testing complex LLM applications, proposes a layered architecture for understanding these challenges, and introduces a lightweight interaction protocol (AICL) to improve testing and quality assurance.

Contribution

It offers a new layered architecture for LLM application testing, identifies core challenges, and proposes a practical, test-oriented communication protocol (AICL) for standardizing testing practices.

Findings

Traditional testing methods are insufficient for LLM layers.

Four fundamental differences cause six core testing challenges.

AICL enables effective communication and testing of AI agents.

Abstract

Applications of Large Language Models~(LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions. Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance. This paper decomposes LLM applications into a three-layer architecture: \textbf{\textit{System Shell Layer}}, \textbf{\textit{Prompt Orchestration Layer}}, and \textbf{\textit{LLM Inference Core}}. We then assess the applicability of traditional software testing methods in each layer: directly applicable at the shell layer, requiring semantic reinterpretation at the orchestration layer, and necessitating paradigm shifts at the inference core. A comparative analysis of Testing AI methods from the software engineering community and safety analysis techniques from the AI community reveals structural disconnects in testing unit abstraction, evaluation metrics, and lifecycle management. We identify four fundamental differences that underlie 6 core challenges. To address these, we propose four types of collaborative strategies (\emph{Retain}, \emph{Translate}, \emph{Integrate}, and \emph{Runtime}) and explore a closed-loop, trustworthy quality assurance framework that combines pre-deployment validation with runtime monitoring. Based on these strategies, we offer practical guidance and a protocol proposal to support the standardization and tooling of LLM application testing. We propose a protocol \textbf{\textit{Agent Interaction Communication Language}} (AICL) that is used to communicate between AI agents. AICL has the test-oriented features and is easily integrated in the current agent framework.

Tables

Table 1. Table 1. Quality assurance value of AI security analysis methods

Analysis dimension	Representative methods	Core role in LLM application testing
Training data privacy	Membership inference (Hu et al., 2022), model inversion (Fredrikson et al., 2015)	Privacy risk assessment and compliance-oriented testing
Model intellectual-property protection	Black-box model extraction (Yue et al., 2021), distillation (Xu et al., 2024), output watermarking (Pan et al., 2024)	IP risk assessment and ownership attribution/tracing
Data poisoning	Poisoning-sample insertion (Zhu et al., 2023), backdoor injection (Yan et al., 2024)	Detection of abnormal behavior during fine-tuning
Deployment and supply-chain security	LoRA weight sideloading (Yao, 2024), KV-cache leakage (Wu et al., 2025), third-party component injection (Zhao et al., 2025)	Multi-tenant inference isolation and supply-chain integrity
Bias and fairness	Multi-attribute bias QA (Parrish et al., 2022), holistic bias prompts (Smith et al., 2022)	Fairness assurance across demographic groups
Adversarial prompt attacks	Jailbreak-prompt generation (Chao et al., 2023), indirect prompt injection (Abdelnabi and Fritz, 2023)	Robustness testing of safety alignment under malicious or adversarial prompts
Output explainability probes	Attention-weight tracing (Artzy and Schwartz, 2024), knowledge-neuron mapping (Dai et al., 2022)	Behavior attribution and debugging in complex scenarios
Watermarking and output tracing	Invisible-text watermarking (SynthID) (Dathathri et al., 2024), robust watermark detection (Tr-GoF) (Li et al., 2024)	Ownership verification and auditing of generated outputs

Table 2. Table 2. Applicability and Collaboration Strategies of Testing Methods Across Challenge Categories

Challenge Category	Traditional Testing	AI-oriented Evaluation	Collaborative Resolution Strategy
Semantic Evaluation and Behavioral Consistency	Exact-match assertions invalid; code coverage inapplicable	Partially applicable (semantic scoring, embedding-based metrics)	Translational: semantic scoring + multi-candidate validation + prompt diversity
Test Data and Robustness in an Open Input Space	Partially applicable (boundary values, equivalence classes)	Partially applicable (adversarial input generation)	Hybrid: distribution-aware data synthesis + anomaly-pattern combination testing
Dynamic State and Observability	Limited state modeling; hard to control environment	Lacks unified state tracking, partial contamination detection	Runtime: end-to-end observability + state replay + semantic contamination detection
Capability Evolution and Regression Risks	Regression testing frameworks available	Limited capability benchmarking	Translational: multi-task capability regression + version-differential analysis
Security and Ethical Compliance	Blind spots in semantic risks; useful for infrastructure security	Specialized risk detection (prompt injection, harmful content)	Preservational: primarily AI security analysis, supplemented by system-level security testing
Multimodal and System-Level Integration	Interface testing applicable but lacks cross-modal scope	Limited cross-modal consistency modeling	Hybrid: end-to-end multimodal pipeline validation + semantic consistency checks

Full text

Large Language Models; Software Testing; AI System Testing; Runtime Testing; Testing Paradigm Evolution

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NY††isbn: 978-1-4503-XXXX-X/2018/06

1. Introduction

Applications of Large Language Models (LLMs) are reshaping the way software systems are constructed. With the advent of technologies such as the Model Context Protocol (MCP)111https://www.anthropic.com/news/model-context-protocol , LLMs have evolved beyond text generation into application systems that support tasks such as retrieval-augmented question answering (Q&A)(Wiratunga et al., 2024) and code generation(Jiang et al., 2024). These systems are no longer simple language generators but complex intelligent software systems (Shim et al., [n. d.]; Wang et al., 2025; McNamara and Marpu, 2025) that integrate retrieval-augmented generation (RAG), tool invocation, multi-turn dialogues, and user interactions.

As LLM applications evolve from single-turn Q&A to interaction modes involving multi-agent orchestration (Mialon et al., [n. d.]), chain-of-thought (CoT) reasoning (Wei et al., 2022), and external API calls, their runtime behavior exhibits greater variability and is more prone to anomalies. Compared to traditional software systems, LLM applications exhibit distinct characteristics (Atf et al., 2025): non-determinism, dynamic behavior, and context dependence. Their outputs arise from the interplay of prompt formulation, contextual state, and model version. Small perturbations in prompt or context can lead to noticeable changes in outputs, a phenomenon loosely analogous to sensitivity to initial conditions in chaos theory (Owotogbe, 2025), while LLM applications are not chaotic systems in the strict scientific sense.

Quality assurance for LLM applications has become both pressing and multifaceted. In traditional software engineering, testing is a cornerstone of quality assurance. Faced with this new paradigm, the question is not whether testing should be abandoned, but how it should be adapted and retained as a primary quality assurance mechanism. Before discussing methodologies, we first analyze the fundamental differences between LLM applications and conventional software. These differences underlie and often give rise to the distinctive testing challenges we face.

1.1. Differences Between LLM Applications and Traditional Software

Traditional software systems are grounded in deterministic algorithmic logic and, under identical execution environments and input, typically produce predictable and repeatable outputs (Promptedmind, 2025; Alphanome AI, 2025). Although non-deterministic behaviors may occasionally arise, for example in cases involving concurrency or reliance on external services, they are usually controllable, analyzable, or mitigated through engineering mechanisms such as synchronization protocols or retry strategies. By contrast, LLM applications reflect shifts: from deterministic computation to probabilistic generation (Wang et al., 2024; Barton, 2023), from purely logical execution to brain-inspired modeling paradigms(Shelton, 2023; Jordan et al., 2019; Zhu et al., 2024a), and from static rule-based systems to dynamically learned behaviors (Zhu et al., 2024b; García-Martín et al., 2021). These shifts manifest in several key aspects:

• •

Open-ended output space: Traditional software typically produces outputs from a predefined, finite set, whereas LLM applications operate over an open natural language space, where the same input may yield multiple semantically valid outputs. This weakens traditional exact-match verification and motivates semantic evaluation methods, such as LLM-as-judge approaches.

• •

Context-dependent behavior: Responses from LLM applications depend not only on the current input but also on dialogue history, system state, and external knowledge. This context dependence exceeds the modeling capacity of traditional testing methods and calls for a shift from stateless function verification to stateful analysis of behavioral sequences.

• •

Emergent and difficult-to-localize capabilities: Certain LLM abilities (e.g., multi-step reasoning, creative writing) emerge during large-scale training and cannot be solely validated by code logic. Their boundaries and reliability are primarily assessed through empirical evaluation rather than formal verification.

• •

System-level composition and complexity: LLM applications integrate multiple subsystems such as retrieval augmentation, tool invocation, and multi-turn interaction, yielding architectures more complex than traditional software. Interactions among these components introduce new failure modes and additional testing requirements.

These fundamental differences help explain when traditional approaches may fail and indicate where innovations in LLM testing are most needed. At its core, testing is an observational activity: a systematic process for examining system behavior under varied conditions. Regardless of methodology, any quality assurance approach for LLM applications should involve such systematic observation. The necessity of testing rests on a simple premise: any deployed system needs to be understood and validated. This requirement applies to both deterministic and probabilistic systems. Indeed, the inherent uncertainty of LLM applications makes systematic behavioral observation more important, not less.

The key question, therefore, is not whether testing is needed, but how methodologies should be adapted or newly designed to align with the characteristics of LLM applications. Completely discarding traditional methods is neither realistic nor necessary. The conventional code and integration logic of an LLM application can still be tested with established techniques, whereas the model inference core requires complementary, AI-specific evaluation approaches. Thus, traditional and novel approaches should coexist, each applied where they are most effective. Our work examines how traditional software testing can be integrated with AI-specific evaluation to address the unique challenges of assuring LLM applications.

1.2. Architectural Model of LLM Applications

The three-layer decomposition we propose in this work is not ad hoc, but a structural pattern consistently observed in both academic research and industry practice. Frameworks such as the 12-Factor Agents blueprint (HumanLayer, 2024), LangGraph (LangChain AI, 2024), and Microsoft AutoGen (Microsoft Research, 2023) exhibit a similar separation among (i) integration and runtime components, (ii) orchestration logic, and (iii) model inference. Comparable distinctions are also documented in surveys of LLMOps and AgentOps (Yehudai et al., 2025; McNamara and Marpu, 2025). This separation reflects clear boundaries in determinism, control, and observability across layers, which correspond to fundamentally different testing focuses. This convergence between independent engineering practices and academic analyses provides a conceptually sound and practically grounded foundation for the subsequent layer-specific discussion of testing. To show system-level behavior in LLM applications and in line with the principle of separation of concerns, we propose three functional layers. We then use this structure to situate the scope of testing methods and organize their coordination.

1. (1)

System Shell Layer: The peripheral support components surrounding the LLM core, with an emphasis on runtime and integration. Typical elements include API interfaces, preprocessing and postprocessing logic, tool invocation modules, human-in-the-loop interfaces, event triggers, and reliability mechanisms such as retries and timeouts. 2. (2)

Prompt Orchestration Layer: Dynamically composes final prompts from user inputs, system instructions, external knowledge, and contextual information. It manages stateful control flow, context compression, and multi-agent handoffs, as exemplified by orchestration frameworks such as LangGraph (LangChain AI, 2024), which support persistent execution state and graph-structured workflows. 3. (3)

LLM Inference Core: Includes the model parameters, inference services, decoding strategies, and safety mechanisms. As the probabilistic generation engine, its outputs depend on prompts, context, and configuration, and it is typically treated as a black box (Shamim and Singhal, 2025).

This structure clarifies the scope and applicability of testing techniques at each layer and underscores the need for cross-layer analysis. This architectural view is independent of any particular testing agenda; subsequent sections leverage it to position and coordinate testing methods. From the System Shell to the LLM Inference Core, behavioral predictability decreases progressively, whereas testing complexity and cost increase correspondingly (Yehudai et al., 2025). For example, unit and integration testing remain effective for shell components such as API interfaces and data preprocessing. The orchestration layer benefits from combining traditional logic validation with semantic quality evaluation, whereas the probabilistic nature of the inference core calls for distinct, probabilistic evaluation paradigms.

Equally important is inter-layer interaction testing. In practice, many faults arise not from individual layers but from data flow and state propagation across layers. For instance, a formatting error in the shell layer may cause the orchestration layer to generate an incorrect prompt. This can then trigger abnormal outputs from the inference core and ultimately degrade system-level behavior (Shamim and Singhal, 2025). Traditional testing methods may overlook such cross-layer error propagation, whereas AI evaluation methods often provide limited modeling of system-wide interactions.

1.3. Research Objectives and Analytical Framework

This paper proceeds along the following objectives:

1. (1)

Applicability of traditional testing (Section 2). Using the three-layer structure, we analyze where classical testing methods remain applicable (e.g., interface validation and shell-layer logic) and where they face limitations. 2. (2)

Survey of AI-oriented evaluations (Section 3). We review AI-oriented evaluation methods in software engineering and assess their alignment with the demands of LLM applications. We also consider complementary AI-safety techniques (privacy attacks, model extraction, data-poisoning detection, and bias assessment) in the context of quality assurance. 3. (3)

Challenge identification and classification (Section 4). Starting from model non-determinism and system composition, we identify the primary challenges in testing LLM applications and organize six core issues into a multi-dimensional view. 4. (4)

Collaborative framework design (Section 5). We outline how multiple methods can be positioned and integrated into a comprehensive assurance workflow. We also propose a lightweight structured protocol for testable LLM agent systems to enable replay and comparison in evaluation.

2. Applicability and Evolution of Traditional Testing Paradigms

Faced with the differences between LLM applications and traditional software, a natural question is whether traditional software testing methods remain effective. While the non-deterministic nature and semantic complexity of LLMs introduce new challenges, LLM applications still consist largely of conventional code. Unit, integration, and performance testing, together with other conventional techniques, remain foundational to software quality assurance. More precisely, traditional testing approaches are not obsolete; they need to be repositioned within the new technical landscape. This section uses the three-layer architectural model to examine the applicability boundaries and evolutionary paths of traditional testing paradigms.

Based on the three-layer architectural model introduced in Section 1.2, we classify the roles of traditional testing methods in LLM applications into three categories:

• •

Retained applications: methods directly applicable to deterministic components.

• •

Translatable adaptations: methods that require redefining test units and objectives.

• •

Integrative innovations: methods that require semantic awareness and extensions for behavioral modeling.

This classification clarifies appropriate usage scenarios and provides a structured roadmap for evolving traditional testing approaches in the LLM era.

2.1. Retainable Role

Within the three-layer architectural model of LLM applications, the System Shell Layer follows the classical input-process-output paradigm and features well-defined functional specifications and state-transition logic. Existing techniques ensure logical correctness and operational stability in this layer. Consequently, traditional testing methods are highly applicable at this level. Specifically:

• •

Unit testing. Verifies the behavior of non-model modules and ensures correct execution of function-level routines such as format parsing and field extraction.

• •

Interface testing. Validates request/response schemas, retry mechanisms, and exception-handling logic. This is especially critical when integrating third-party LLM APIs (e.g., OpenAI, Claude).

• •

Integration testing. Checks cross-component interactions and ensures end-to-end functionality from user input through the retrieval component to LLM inference.

• •

Performance and stress testing. Evaluates service-level metrics such as response latency, memory footprint, and GPU utilization under high concurrency, multi-turn interactions, and large-input scenarios.

These traditional methods provide a foundation for engineering LLM applications. They directly support reliability and maintainability and help establish a stable operational baseline.

2.2. Translatable Role

Although the core concepts of traditional testing are not directly applicable to LLM applications in their original form, the underlying principles of adequate coverage, behavioral consistency, and regression prevention remain valuable. In LLM applications, these principles should be reinterpreted using semantic behavior modeling. For example, assertion-based testing of the form “expected == actual” faces fundamental challenges in LLM applications because outputs are natural-language sequences characterized by multiplicity and subjectivity. Boolean assertions should be augmented or replaced by mechanisms such as LLM-based scoring, semantic-similarity measures, or multi-candidate comparison:

• •

LLM-as-judge scoring: this approach is easy to implement and widely used, but it requires monitoring scorer consistency and potential bias.

• •

Semantic-similarity evaluation: compute cosine similarity using models such as BERT or Sentence-Transformers. A key challenge is setting appropriate similarity thresholds.

• •

Multi-candidate voting: generate multiple candidate outputs and use an expert-agent system to select the best answer.

Translating coverage testing entails shifting from logical paths to semantic paths. Traditional notions such as path and branch coverage have limited validity given the black-box nature of LLM inference. Prompts define linguistic rather than logical paths. Accordingly, new coverage metrics should be defined:

• •

Prompt-embedding coverage. Map prompts to a high-dimensional embedding space and use clustering to ensure that test cases cover diverse semantic regions.

• •

Semantic-category coverage. Define a compositional space of prompt templates to ensure adequate coverage of task types, linguistic styles, and complexity levels.

• •

Output-behavior diversity. Use entropy-based indicators to assess the richness of output distributions and reduce mode collapse.

Regression testing should also be reinterpreted in terms of behavioral consistency. Traditional output-equivalence checks are often unreliable, because identical inputs can yield divergent outputs due to context, caching, or versioning. Therefore, new strategies for evaluating behavioral consistency are needed. For example, in code-generation stability testing, using 100 semantically equivalent function descriptions (e.g., “sort a list” versus “arrange list elements in order”), the model is expected to achieve at least 95% functional correctness with acceptable stylistic variation.

This translational strategy embodies the core idea of semantic behavior modeling, which shifts the test unit from static code paths to observable linguistic behavior patterns, including output distributions, semantic categories, and context-driven variation.

2.3. Integrative Role

Traditional testing methods primarily focus on functional correctness, whereas the testing objectives for LLM applications also include content plausibility, contextual consistency, factual accuracy, bias, and safety. Achieving these goals requires incorporating methods from natural language processing, AI safety, and human-computer interaction testing to construct a cross-disciplinary, integrated framework.

For output robustness, traditional approaches often overlook linguistic variation and ambiguity. In contrast, LLM testing should employ semantics-preserving prompt rewrites to evaluate output variance and stability across paraphrases. Factual and knowledge verification is not a primary focus of traditional testing. LLM testing should incorporate retrieval-augmented checkers and time-sensitive datasets to assess hallucinations, staleness, and factual drift. Testing contextual-state consistency involves constructing context-state trees and analyzing prompt-behavior consistency and transition patterns across alternative context paths, which may exceed the modeling capacity of traditional user-scenario testing. Moreover, for domain-specific LLM applications, constructing a knowledge graph of real-world facts may serve as a way to approximate semantic test coverage. However, building such a knowledge base is nontrivial.

This category of testing gives rise to a new research direction that aims to build a complete testing loop spanning semantic generation, behavioral observation, and contextual memory. This integrative role indicates that LLM application testing is not merely an extension of existing techniques but also entails a transformation in testing philosophy and objectives.

Traditional software testing remains valuable and provides a foundation for quality assurance in LLM applications. However, as discussed earlier, traditional approaches face notable limitations in both the translatable and integrative roles, particularly in semantic evaluation, behavior modeling, and contextual-state management. These challenges motivate ongoing development of testing methods specifically tailored to AI systems.

It is therefore worth examining whether emerging AI-oriented testing methodologies can effectively address the gaps left by traditional testing. In the following section, we examine the current landscape and limitations of AI testing research and assess how closely these efforts align with the needs of LLM applications.

3. Current Landscape and Capability Boundaries of AI Testing Research

Both the software engineering and AI research communities are actively exploring new testing theories and methodologies. However, an open question is whether these efforts address the quality-assurance needs of LLM applications and whether structural disconnects exist between the approaches taken by the two communities.

Current research on AI testing has evolved into two largely independent directions: one is Testing AI from a software engineering perspective, and the other is security analysis techniques rooted in AI safety. The former emphasizes functional correctness and engineering quality, whereas the latter targets model security risks and ethical considerations.

Category 1: Testing AI (software engineering perspective)

Researchers in the software engineering (SE) community consider AI models as software components and focus on functional correctness, reliability, and engineering quality (Zhang et al., 2020). The core objectives are to verify that AI systems operate as expected, to detect functional flaws, and to evaluate system performance. This line of work builds on and extends traditional theories, methods, and tools in software testing.

Category 2: AI security analysis (AI research perspective)

Researchers in the AI community focus on safety, privacy, fairness, and ethical risks associated with AI models. Their goals are to identify vulnerabilities, privacy leaks, algorithmic bias, and adversarial weaknesses (Hu et al., 2021). This line of work builds on subfields such as machine learning security, privacy-preserving learning, and algorithmic fairness.

While both categories have made significant progress in their respective domains, each faces notable limitations when addressing the full complexity of LLM applications. More critically, differences in focus and methodology between the two communities have led to a structural disconnect in addressing quality assurance for LLM applications.

3.1. Testing AI from SE

AI testing research in the software engineering community has primarily focused on extending established theories and methodologies from traditional software testing to AI systems. The central idea is to treat AI models as software components with distinctive properties. Under this paradigm, mainstream Testing AI approaches generally follow three technical pathways:

(1) Input perturbation and robustness testing

This category draws on mutation testing by introducing character-level, word-level, or syntactic perturbations to inputs to generate adversarial or variant test cases. The objective is to verify whether outputs remain semantically consistent. This assumes that invariant input semantics correspond to invariant output semantics, using input stability as a proxy for output robustness. These methods are mainly applicable to static-input tasks such as text classification, sentiment analysis, and single-turn Q&A. However, they are limited for multi-turn context, state evolution, external tool calls, and prompt chaining, and they offer little support for modeling complex prompt assembly and contextual memory.

DeepXplore (Pei et al., 2017) pioneered white-box testing of neural networks, using differential testing to identify inconsistent DNN behaviors. Tools such as TensorFuzz (Odena et al., 2019) and DeepHunter (Xie et al., 2019) apply coverage-guided fuzzing. Frameworks such as CheckList (Ribeiro et al., 2020) offer systematic behavioral testing for NLP models. Tools including TextFooler (Jin et al., 2020), DeepWordBug (Gao et al., 2018), and TextBugger (Li et al., 2019) focus on adversarial text generation. The DeepMutation series (Ma et al., 2018) extends mutation testing to deep learning. Recent open-source frameworks such as DeepEval222https://github.com/confident-ai/deepeval attempt to integrate traditional testing techniques with AI-safety analyses, offering unified interfaces and metrics. However, they still face limitations in modeling system-level behavior and in reducing reliance on external evaluators.

(2) Metamorphic testing

Metamorphic testing333https://en.wikipedia.org/wiki/Metamorphic_testing defines relations between inputs and outputs, known as metamorphic relations (MRs), to evaluate behavioral consistency when ground-truth answers are unavailable. Its core value is enabling large-scale automated test generation and supporting multidimensional evaluation, including diversity, robustness, and fairness.

In practice, metamorphic testing has shown promise across several domains. For example, DeepRoad (Zhang et al., 2018) applies GAN-based image transformations for autonomous-driving system testing, and in NLP testing researchers use language-preserving transformations such as synonym substitution and syntactic rewriting to verify consistency. Tools such as METAL (Hyun et al., 2024) and MORTAR (Guo et al., 2024), together with prompt-level semantic metamorphic testing, represent subsequent developments of this approach. Nevertheless, metamorphic-testing methods face clear limitations. The design of metamorphic relations often relies on manual heuristics, increasing complexity and limiting scalability to the intricate prompt paths, tool-call sequences, and memory-dependent behaviors present in LLM applications.

(3) Neural probing and coverage modeling

Neural probing and coverage modeling seek to estimate test-sample coverage by analyzing neuron activations, attention patterns, or distributional clustering. The goal is to guide test generation and behavioral detection. The core idea is to transfer coverage concepts from traditional software testing to deep neural models to estimate test adequacy when ground-truth labels are unavailable.

This line of work has produced several technical branches: Structural coverage focuses on neuron and branch coverage as well as boundary metrics (Usman et al., 2023). Behavioral coverage, such as Surprise Adequacy (Kim et al., 2023), measures adequacy based on behavioral distributions. Symbolic verification methods such as Reluplex (Katz et al., 2017) employ SMT solvers for formal verification of neural networks. Concolic testing (Sun et al., 2018) extends symbolic execution techniques to DNNs. However, these methods face practical obstacles. The internal states of deep models are high-dimensional and lack standardized representational frameworks, and, most critically, correlations between coverage and actual functional defects are often weak. This limits their applicability in real-world testing.

3.2. AI Security Analysis Methods

Security analysis methods in the AI research community arise from investigations into the intrinsic characteristics of machine learning models. These methods primarily focus on internal security risks, ethical considerations, and broader social implications (Hu et al., 2021; Rahman et al., 2023). Originally, the goal was to assess properties (security, privacy, fairness, and robustness) that differ fundamentally from the functional-correctness verification emphasized in the software engineering community (Das et al., 2025; Yao et al., 2024).

From the perspective of LLM application testing, these methods constitute a core component of the quality-assurance landscape, especially for addressing semantic safety, behavioral consistency, and ethical risk that traditional functional testing may not cover. Table 1 summarizes the categories and value of AI security analysis methods for testing LLM applications.

Layer-Specific Applicability of AI Security Analysis Methods

Although AI security analysis methods are a core component of LLM application testing, their applicability is mostly confined to specific architectural layers. Traditional software testing primarily targets the system shell layer, focusing on functional correctness and performance stability. In contrast, AI security analysis methods are designed to address the unique challenges posed by the prompt orchestration layer and the LLM inference core, namely, semantic safety, behavioral consistency, and ethical risks, that are difficult to capture through traditional testing.

Current research suggests that AI security analysis methods are largely focused on the third layer (the LLM inference core) emphasizing internal model property evaluation. In contrast, the second layer, which involves complex prompt orchestration and context-state management, receives comparatively little attention. This imbalance in research emphasis leads to adaptation challenges when applying these methods in practical system-level scenarios.

3.3. Structural Disconnection Between the Two Methodological Paradigms

The divergence in methodologies, evaluation metrics, and technical pathways between the two communities has led to a structural disconnection in addressing the complex requirements of LLM applications. This disconnection reflects not only limitations of individual methods but, more importantly, the lack of unified theoretical frameworks and collaborative mechanisms.

Structural limitations of Testing AI approaches

The software engineering community is accustomed to testing deterministic systems, and its testing theories rest on assumptions of reproducibility, controllability, and predictability. However, the non-deterministic characteristics of LLM applications are in tension with these assumptions. We identify four systemic mismatches between Testing AI approaches and the needs of LLM applications:

1. (1)

Mismatch in test unit abstraction. AI testing typically treats the “input sample” as the test unit, while the behavior of an LLM application system is a composite triggering structure composed of “prompt chaining + state memory + multi-agent coordination”. In real systems, agent tasks often involve sequences such as “goal decomposition → tool planning → execution feedback → state update → behavioral recursion”, which go far beyond the expressive scope of current testing paradigms. 2. (2)

Incompatibility between behavioral modeling assumptions and system execution paths. Existing tests are mostly based on a short-path model assumption of “input → output”, while actual systems follow a heterogeneous pipeline consisting of “prompt + retrieval + tool invocation + multi-turn state”. In practice, prompt perturbation tests are often neutralized by concatenation mechanisms, filtering rules, system prompts, or session memory, limiting their effectiveness. Moreover, behaviors of key components such as prompt orchestrator, and tool states are often unobservable. Current testing methods cannot trace the generation path, resulting in unexplainable test outcomes, undiagnosable errors, and unreproducible regressions. 3. (3)

Lack of multi-dimensional semantic evaluation metrics. Commonly used metrics such as BLEU, coverage, and clustering distance offer limited discriminative power. These methods suffer from major deficiencies and they lack a behavioral classification system and dimension-specific metrics, making feedback-driven optimization difficult. 4. (4)

Missing lifecycle testing pipeline. Most research focuses on pre-deployment testing, while real-world systems undergo continuous evolution, such as prompt strategy updates, LoRA fine-tuning iterations, and RAG index replacements; There is a lack of CI/CD-integrated mechanisms for version comparison testing, behavioral difference detection, and semantic regression verification; Test sets cannot incrementally grow from real-world anomalies and quickly become obsolete.

Adaptation Difficulties of AI Security Analysis Methods

Existing AI security analysis methods face notable adaptation challenges when applied to real-world LLM systems, primarily due to differences between research assumptions and production environments. Many of these methods rest on an idealized premise that treats the model as the sole object of study, emphasizing internal properties and theoretical bounds while overlooking interaction effects and state dependencies in complex system environments. In practice, prompt data in LLM applications typically flows through multi-stage transformation pipelines, which often undermines the effectiveness of attacks designed for static, single-turn inputs. Moreover, features such as memory mechanisms, session-state caching, and multi-agent collaboration cause responses to depend not only on the current input but also on historical outputs, forming intricate indirect causal chains. This dynamic state evolution can expose and amplify security risks over multi-turn interactions. However, traditional static evaluation, limited to a single point in time, may fail to capture this accumulation of risk. As a result, evaluation outcomes derived from isolated experiments may not generalize well to production environments characterized by multi-component coordination, shared state, and concurrent access.

Looking ahead, the capability boundaries and disconnects outlined above define the problem space we address next: Section 4 organizes testing pain points into system-level challenge categories grounded in the architectural view introduced in Sections 1.2.

4. Core Challenges in Testing LLM Applications

Building on the fundamental differences in Section 1.1, the three-layer architectural view in Section 1.2, and the capability boundaries identified in Section 3, we consolidate the problem space into system-level challenge categories for testing LLM applications. Each category is defined at a granularity beyond a single prompt–response pair, aligned with the architecture, and motivated by the issues discussed above.

4.1. Challenges

4.1.1. Semantic Evaluation and Behavioral Consistency

Outputs of LLM applications lie in an open-ended natural-language generation space, where a single input may admit multiple semantically correct answers. Traditional exact-match verification methods are therefore insufficient on their own in this context. Moreover, conventional coverage metrics (e.g., statement or branch coverage) do not capture the breadth of the semantic behavior space, leaving few reliable quantitative measures of behavioral adequacy.

Compounding the difficulty, LLMs exhibit marked sensitivity to superficial linguistic variation, such that minor, semantically equivalent rephrasings can trigger substantial fluctuations in output quality. As a result, testing efforts struggle to define a unified correctness standard, accurately measure coverage, and predict system behavior in production, which can affect user experience and system stability. Addressing these challenges calls for a shift from static path coverage to semantic behavior modeling, combined with techniques such as LLM-as-judge and semantic-similarity measures to evaluate output consistency and stability.

4.1.2. Test Data and Robustness in an Open Input Space

The input space of LLM applications is open and semantically unbounded. Real-world user inputs often contain typos, dialectal expressions, emotional language, or implicit assumptions, making full test coverage impractical. Furthermore, LLM boundary conditions are not simple numerical or format constraints but semantically defined extremes. Such atypical inputs can cause performance degradation, logical incoherence, or hallucinations.

Constructing test sets that both represent real-world distributions and cover abnormal scenarios is costly and prone to rapid obsolescence. The combinatorial effects of different anomaly types can grow rapidly, making traditional boundary-value analysis insufficient. Therefore, robustness assessment calls for distribution-aware data generation and combinatorial testing of anomaly patterns to maintain ecological validity in evaluation.

4.1.3. Dynamic State and Observability

Many LLM systems are stateful. Historical context, key-value (KV) caches, and conversational memory can propagate across turns, leading to behavioral carryover and violations of user isolation. At the same time, LLM outputs are influenced by multiple layered components, including retrieval modules, prompt templates, and reasoning engines. However, many systems lack a unified tracing and observability framework, which makes failure reproduction and root-cause diagnosis challenging.

Such state dependence and limited visibility undermine the basis for debugging, regression testing, and performance optimization. Mitigation strategies include introducing end-to-end observability frameworks with unified trace IDs and reconstructable context states, multi-turn conversation-replay mechanisms, and cross-component data-flow monitoring tools.

4.1.4. Capability Evolution and Regression Risks

After training, a model’s internal knowledge is effectively fixed and can become outdated over time. Task-specific fine-tuning can enhance targeted capabilities while potentially degrading overall performance. Without a standardized multi-task regression testing framework, capability degradation or regression is often detected only weeks or months after deployment.

These risks affect not only time-sensitive applications (e.g., news, finance, and legal domains) but also lead to unpredictable behavioral differences in multi-version deployments. Addressing these risks calls for establishing multidimensional capability benchmarks, coupled with continuous regression testing and version-differential analysis, to enable early warning of knowledge obsolescence and capability drift.

4.1.5. Security and Ethical Compliance Challenges

LLMs can be sensitive to adversarial prompts (jailbreaks) and covert malicious instructions. Attackers may bypass safety mechanisms through role-playing, indirect prompt injection, or other prompt-manipulation techniques. Divergent cultural and regulatory definitions of “harmful” complicate cross-jurisdictional compliance.

Latent risks include harmful content embedded in figurative language, culturally sensitive topics, and passively triggered malicious instructions. Legal accountability is often ambiguous, which calls for multi-layer safety defenses at the model, application, and user levels, together with dynamic safety evaluation and localized compliance strategies.

4.1.6. Multimodal and System-Level Integration Challenges

LLM applications increasingly integrate capabilities such as image understanding, speech recognition, video analysis, and multi-agent collaboration, yielding complex multimodal systems. Errors in one modality can be amplified or masked as they propagate through the pipeline, making them difficult to detect in single-module tests.

Meanwhile, long-context processing, repeated tool invocations, and multi-agent coordination can introduce nonlinear increases in inference latency and resource consumption. Standardized methods for cross-modal consistency verification are lacking, and overall system performance is jointly influenced by semantic complexity, interaction depth, and call-chain length, complicating capacity planning and cost control. Addressing these issues calls for cross-modal consistency-checking frameworks and semantic-aware performance-testing methods, together with early-stage design mechanisms to balance performance, accuracy, and cost.

4.2. The Inevitable Shift in Testing Paradigms

This analysis indicates that LLM testing challenges are not merely technical or procedural; they signal a shift in testing philosophy:

• •

From deterministic validation to probabilistic evaluation: prioritize semantic equivalence, calibrated confidence intervals, and robustness analysis over exact-match checks.

• •

From static coverage to dynamic exploration: adopt risk-driven, distribution-aware, and adversarial data generation in place of fixed test sets.

• •

From one-time validation to continuous assurance: employ continuous monitoring, drift detection, and runtime regression to address model evolution and state-dependent behavior.

• •

From single method to collaborative ecosystem: integrate semantic evaluation, robustness probing, multimodal consistency verification, and performance and cost optimization into a coordinated testing framework.

These shifts are not incremental improvements; they redefine the aims and instruments of testing and serve as guiding principles for assessing the applicability and evolution of methods in the sections that follow.

5. Collaborative Testing Framework Design

Based on the four paradigm shifts, we first outline a collaborative testing framework that integrates the strengths of diverse testing methodologies. We then introduce AICL, a lightweight structured protocol that operationalizes this framework by ensuring semantic precision, observability and provenance, deterministic replay, context isolation, and built-in evaluation hooks for fair comparison.

5.1. Collaboration Strategies

Given the six challenge categories consolidated above, no single testing method is sufficient on its own. Traditional testing remains effective for deterministic components but lacks semantic evaluation. AI analysis methods introduce innovations in behavioral modeling but remain underdeveloped for system-level engineering (Yuan et al., 2023). AI security analysis excels at risk identification yet is often disconnected from real-world operational contexts (Liu et al., 2023b; Wei et al., 2023). Each approach exhibits distinct strengths and blind spots.

The core of a collaborative testing framework is to translate the four paradigm shifts into actionable testing strategies (Section 4.2). This requires a unified quality-assurance system that coherently integrates the strengths of different methods while mitigating their limitations. Building on the six-category analysis, we illustrate the method applicability boundaries and design effective collaboration strategies.

From the mappings in Table 2, it follows that different challenges call for different collaborative strategies. Based on similarities in resolution approaches, the six challenge categories can be addressed via four collaborative modes.

1. Preservational application: extending the strengths of dominant methods

In some categories, one testing approach is primary and the other plays a complementary role. For example, system-level performance concerns in the multimodal and system-level integration category can often be addressed with mature, traditional techniques for performance, stress, and load testing (ISO/IEC, 2023). Although LLM inference patterns differ from traditional software execution, bottlenecks, memory leaks, and concurrency issues can often be diagnosed with established tools.

By contrast, security and ethical compliance relies primarily on AI security analysis. LLM-specific threats (such as prompt injection, generation of harmful content, and privacy leaks (Liu et al., 2023b; Greshake et al., 2023; OWASP, 2025)) require specialized detection techniques. Traditional security testing continues to play a supporting role at the infrastructure level (e.g., vulnerability scanning, access-control verification).

2. Translational adaptation: semantic reconstruction of testing concepts

Some challenges require reinterpreting traditional testing concepts within the semantic space of LLMs. The semantic evaluation and behavioral consistency challenge exemplifies this shift: although binary expected-versus-actual checks are inadequate, the objective of correctness validation remains. Assertions should be reformulated as multidimensional semantic evaluations that employ LLM-as-judge scoring or embedding-based semantic-similarity measures (Zheng et al., 2023; Liu et al., 2023a).

Similarly, the capability evolution and regression risks challenge extends regression testing to multi-task capability assessment. Rather than detecting functional defects, we benchmark reasoning, comprehension, and generation and compare performance across model versions to identify degradation.

3. Integrative innovation: deep synthesis of cross-domain methods

Some challenges require true methodological synthesis. For test data and robustness in an open input space, boundary-value analysis and equivalence partitioning should be combined with adversarial-prompt generation and mutation-based perturbations (Yang et al., 2024; Xhonneux et al., 2024). Multimodal and system-level integration requires extending interface testing to end-to-end multimodal pipelines and coupling it with cross-modal semantic-consistency verification (Abootorabi et al., 2025; Han et al., 2024). In semantic evaluation and behavioral consistency, robustness testing benefits from integrating semantics-preserving equivalence transformations with AI-driven adversarial-perturbation techniques (Weng, 2023; Zou et al., 2023).

4. Runtime extension: continuous assurance beyond deployment

Certain categories, such as dynamic state and observability, indicate that many critical issues (e.g., state contamination and knowledge drift) surface only in production. Static pre-deployment testing is not sufficient on its own; runtime quality assurance is needed.

In this mode, traditional monitoring (system-level metrics, anomaly detection) is combined with AI-oriented methods (semantic-quality monitoring, behavior-drift detection) to maintain continuous quality. For example, conversation-state isolation can be reinforced with semantic-contamination detection to preserve user separation and reduce subtle leakage.

5.2. AICL: A Structured Protocol for Testable LLM Application

While free-form natural language facilitates rapid prototyping, it hinders testability: it is ambiguous, hard to parse deterministically, and lacks machine-readable context. This is a key reason why many existing testing techniques do not directly carry over. Inspired by the work (Xing et al., 2025), we therefore introduce the Agent Interaction Communication Language (AICL), a schema-driven, transport-independent protocol that (i) enforces semantic precision via typed messages, (ii) encodes observability and provenance into message metadata, and (iii) guarantees replayability through canonical encodings and explicit priors for probabilistic reasoning. AICL operationalizes the four paradigm shifts (Section 4.2) by making probabilistic evaluation, dynamic exploration, runtime assurance, and method collaboration measurable and automatable. AICL is not intended for end users. To make it practical for developers, auxiliary tooling (e.g., NL to AICL translators) is needed.

Protocol Overview.

Each AICL message has a compact text form and a canonical Concise Binary Object Representation444https://en.wikipedia.org/wiki/CBOR (CBOR) encoding:

[TYPE: CONTENT | METADATA]

Core Message Types.

AICL defines a small set of canonical message types that serve as building blocks for agent interaction. Each type is motivated by the need for semantic clarity, observability, and testability:

• •

HELLO — session initialization and handshake; declares agent identity, capabilities, and version, enabling environment comparison in replay.

• •

QUERY — issues a request to another agent or tool, carrying input parameters and context identifiers in a structured form.

• •

PLAN — expresses a multi-step reasoning or execution plan, with explicit ordering and dependency markers.

• •

FACT/FACTS — states known information or environmental conditions, optionally annotated with confidence scores for probabilistic evaluation.

• •

RESULT — returns the output corresponding to a QUERY or PLAN, including data payloads and evaluation metadata.

• •

ERROR — standardized error reporting with error codes, context, and optional recovery hints, facilitating systematic failure analysis in testing.

• •

MEMORY.STORE — explicitly stores state or information, tagged with scope and context for controlled retrieval.

• •

MEMORY.RECALL — retrieves previously stored information, supporting reproducibility and context isolation during tests.

• •

COORD.DELEGATE — delegates a subtask to another agent or external tool, supporting collaborative multi-agent scenarios.

• •

REASONING.(START|STEP|COMPLETE) — marks stages in structured reasoning traces, enabling fine-grained capture of intermediate thoughts for later verification.

Each message is accompanied by core metadata; all fields are required unless noted. The field of is required on response-type messages, and reasoning_trace, cost, latency, sig, and cap are optional.

• •

id: unique identifier for traceability

• •

ts: timestamp of emission

• •

ver: protocol version

• •

cid: canonical conversation identifier for replay

• •

ctx: explicit context scope (ensuring isolation between tests)

• •

model_version: agent model version for comparability

• •

conf: confidence score reported by the model

• •

priors: explicitly declared probabilistic assumptions

• •

space: domain or task space of the message

• •

of: correlation pointer to the upstream message id that this message responds to (e.g., a RESULT referring to its QUERY); typically absent on initial QUERY

• •

reasoning_trace: structured markers (e.g., REASONING.START—STEP—COMPLETE) and trace payloads for capturing intermediate reasoning for later verification and analysis

• •

cost/latency: resource- and time-related annotations for performance baselining (e.g., token usage, tool/runtime latency, wall-clock duration)

• •

sig/cap: integrity and capability attestations for runtime assurance (e.g., cryptographic integrity checks and declared/attested capabilities of the agent or environment)

Example: Query/Result Pattern.

ΨΨ[QUERY: tool:weather_now{location:"Shanghai"} ΨΨ| id:u!q1, ts:t(2025-08-15T02:00:00Z), ver:"1.2.0", ctx:[u!conv123]] ΨΨ ΨΨ[RESULT: {data:{temp_c:29, cond:"rain"}, schema:"tool:weather_now/1"} ΨΨ| id:u!r1, of:u!q1, conf:0.93, model_version:"gpt-5-20250801", ts:t(...)] Ψ

This example shows how a structured QUERY generates a corresponding RESULT, with explicit schema, confidence, and provenance metadata. Unlike natural language responses, the structured fields make it possible to evaluate correctness, latency, and reliability automatically.

Test-Oriented Features and Testing Benefits.

AICL is designed explicitly for test harnesses. Its protocol-level features map directly to common challenges in LLM application testing:

• •

Deterministic Replay and Reproducibility: Canonical CBOR encoding, together with identifiers such as cid, priors, and model_version, ensures that past runs can be retrieved and replayed exactly.

• •

Context Isolation: Explicit ctx and MEMORY.* operations prevent hidden state leakage, enabling controlled experiments and differential testing.

• •

Built-in Evaluation Hooks: Structured fields such as conf, reasoning_trace, cost/latency annotations, and typed RESULT payloads allow automated verdicts, schema-based validation, and performance baselining without ambiguity.

• •

Failure Diagnosis and Safety: Standardized ERROR codes, reasoning trace markers, and integrity checks (sig/cap) facilitate systematic error analysis and runtime assurance.

• •

Runtime Reusability: Production AICL logs can be ingested directly into offline regression suites, supporting continuous test–deploy–replay cycles.

Integration with the Collaborative Framework.

AICL provides the structured substrate for the four collaborative modes of agent testing:

• •

Preservational: messages act as stable, machine-verifiable interfaces, allowing classical unit and integration tests to be re-applied.

• •

Translational: semantic assertions can be applied directly to structured RESULT fields, avoiding the ambiguity of free-text evaluation.

• •

Integrative: tool calls, prompt templates, and retrieval events are normalized as QUERY/RESULT chains, enabling consistent coverage metrics across heterogeneous systems.

• •

Runtime Extension: production interactions expressed in AICL are natively reusable in regression testing, enabling continuous monitoring and runtime quality assurance.

IBM recently introduced the Prompt Declaration Language (PDL) (Heller et al., 2024), which, like AICL, seeks to address the fragility of natural language interactions in LLM applications. PDL, as a YAML-based declarative language, emphasizes structured and maintainable prompt design, while AICL, as an interaction protocol, focuses on testing and verification through standardized message types, context isolation, and replayability. Thus, PDL operates mainly at the design layer, whereas AICL targets the execution and testing layer, making them complementary.

6. Conclusion

This paper is motivated by real-world quality-assurance demands of LLM applications and offers an in-depth analysis of the challenges and opportunities in this emerging field. A systematic review of the current state of AI testing reveals a structural disconnection between the software engineering and AI research communities. By analyzing fundamental differences between LLM applications and traditional software, we identify six core categories of testing challenges. By decomposing the LLM architecture into three functional layers, we discuss and analyze the applicability and evolutionary trajectory of conventional testing methods. We propose a collaborative testing framework and an agent interaction communication language (AICL), offering a practical, systematic approach to quality assurance for LLM applications.

A key insight is that LLM application testing is neither a mere extension of traditional software testing nor a straightforward application of AI-security techniques. Rather, it is a systems-level engineering task that integrates multiple disciplinary strengths to construct a new quality-assurance paradigm. In the face of probabilistic generation, open-ended semantics, and dynamic evolution, what is required is not only technical innovation but also a shift in testing philosophy. As large language models continue to evolve and their applications proliferate, the field of LLM application testing will continue to encounter new challenges and opportunities. The analytical framework and collaborative testing architecture presented in this work provide a theoretical foundation for advancing this field, but their effectiveness should be validated and refined through empirical research, engineering practice, and cross-domain collaboration. Sustained progress will require collaboration among researchers and practitioners in software engineering, artificial intelligence, systems operations, and beyond. Advances in theory, methods, and engineering practice should proceed together to build a new quality-assurance system suited to the era of intelligent software.