From Dispersion to Attraction: Spectral Dynamics of Hallucination Across Whisper Model Scales

TL;DR

This paper introduces the Spectral Sensitivity Theorem to predict phase transitions in large ASR models' spectral dynamics, linking model size to hallucination behaviors and spectral regimes.

Contribution

It proposes a novel spectral theory predicting regime shifts in deep networks and validates it through eigenspectra analysis of Whisper models under stress.

Findings

Intermediate models show 13.4% collapse in Cross-Attention rank.

Large models enter a regime with 2.34% Self-Attention rank compression.

Spectral regimes correlate with hallucination behaviors in ASR models.

Abstract

Hallucinations in large ASR models present a critical safety risk. In this work, we propose the \textit{Spectral Sensitivity Theorem}, which predicts a phase transition in deep networks from a dispersive regime (signal decay) to an attractor regime (rank-1 collapse) governed by layer-wise gain and alignment. We validate this theory by analyzing the eigenspectra of activation graphs in Whisper models (Tiny to Large-v3-Turbo) under adversarial stress. Our results confirm the theoretical prediction: intermediate models exhibit \textit{Structural Disintegration} (Regime I), characterized by a $13.4\%$ collapse in Cross-Attention rank. Conversely, large models enter a \textit{Compression-Seeking Attractor} state (Regime II), where Self-Attention actively compresses rank ($-2.34\%$) and hardens the spectral slope, decoupling the model from acoustic evidence.