Semantic Invariance in Agentic AI

TL;DR

This paper introduces a metamorphic testing framework to evaluate the semantic invariance of large language models acting as reasoning agents, revealing that larger models are not necessarily more robust to input variations.

Contribution

We develop a systematic testing method using semantic-preserving transformations to assess LLM reasoning robustness across multiple models and domains.

Findings

Smaller models like Qwen3-30B-A3B show higher robustness (79.6%) than larger models.

Model scale does not correlate with increased robustness.

Semantic invariance varies significantly across models and transformations.

Abstract

Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance. Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility.