# Rethinking Testing for LLM Applications: Characteristics, Challenges, and a Lightweight Interaction Protocol

## TL;DR

This paper analyzes the unique challenges of testing complex LLM applications, proposes a layered architecture for understanding these challenges, and introduces a lightweight interaction protocol (AICL) to improve testing and quality assurance.

## Contribution

It offers a new layered architecture for LLM application testing, identifies core challenges, and proposes a practical, test-oriented communication protocol (AICL) for standardizing testing practices.

## Key findings

- Traditional testing methods are insufficient for LLM layers.
- Four fundamental differences cause six core testing challenges.
- AICL enables effective communication and testing of AI agents.

## Abstract

Applications of Large Language Models~(LLMs) have evolved from simple text generators into complex software systems that integrate retrieval augmentation, tool invocation, and multi-turn interactions. Their inherent non-determinism, dynamism, and context dependence pose fundamental challenges for quality assurance. This paper decomposes LLM applications into a three-layer architecture: \textbf{\textit{System Shell Layer}}, \textbf{\textit{Prompt Orchestration Layer}}, and \textbf{\textit{LLM Inference Core}}. We then assess the applicability of traditional software testing methods in each layer: directly applicable at the shell layer, requiring semantic reinterpretation at the orchestration layer, and necessitating paradigm shifts at the inference core. A comparative analysis of Testing AI methods from the software engineering community and safety analysis techniques from the AI community reveals structural disconnects in testing unit abstraction, evaluation metrics, and lifecycle management. We identify four fundamental differences that underlie 6 core challenges. To address these, we propose four types of collaborative strategies (\emph{Retain}, \emph{Translate}, \emph{Integrate}, and \emph{Runtime}) and explore a closed-loop, trustworthy quality assurance framework that combines pre-deployment validation with runtime monitoring. Based on these strategies, we offer practical guidance and a protocol proposal to support the standardization and tooling of LLM application testing. We propose a protocol \textbf{\textit{Agent Interaction Communication Language}} (AICL) that is used to communicate between AI agents. AICL has the test-oriented features and is easily integrated in the current agent framework.