Evaluating Semantic Fragility in Text-to-Audio Generation Systems Under Controlled Prompt Perturbations

TL;DR

This paper evaluates the robustness of text-to-audio models under controlled linguistic variations, revealing that larger models are more consistent but all exhibit fragility during semantic-to-audio conversion.

Contribution

It introduces a novel framework for assessing semantic robustness in text-to-audio systems across multiple representational levels.

Findings

Larger models like MusicGen-large show higher semantic consistency.

All models exhibit divergence in acoustic and temporal features despite high embedding similarity.

Fragility mainly occurs during semantic-to-acoustic realization, not in multi-modal embedding alignment.

Abstract

Recent advances in text-to-audio generation enable models to translate natural-language descriptions into diverse musical output. However, the robustness of these systems under semantically equivalent prompt variations remains largely unexplored. Small linguistic changes may lead to substantial variation in generated audio, raising concerns about reliability in practical use. In this study, we evaluate the semantic fragility of text-to-audio systems under controlled prompt perturbations. We selected MusicGen-small, MusicGen-large, and Stable Audio 2.5 as representative models, and we evaluated them under Minimal Lexical Substitution (MLS), Intensity Shifts (IS), and Structural Rephrasing (SR). The proposed dataset contains 75 prompt groups designed to preserve semantic intent while introducing localized linguistic variation. Generated outputs are compared through complementary spectral, temporal, and semantic similarity measures, enabling robustness analysis across multiple representational levels. Experimental results show that larger models achieve improved semantic consistency, with MusicGen-large reaching cosine similarities of 0.77 under MLS and 0.82 under IS. However, acoustic and temporal analyses reveal persistent divergence across all models, even when embedding similarity remains high. These findings indicate that fragility arises primarily during semantic-to-acoustic realization rather than multi-modal embedding alignment. Our study introduces a controlled framework for evaluating robustness in text-to-audio generation and highlights the need for multi-level stability assessment in generative audio systems.