Spectral Geometry for Deep Learning: Compression and Hallucination Detection via Random Matrix Theory

TL;DR

This paper introduces a spectral geometry framework using random matrix theory to improve neural network reliability and efficiency through hallucination detection and model compression.

Contribution

It presents EigenTrack for real-time hallucination detection and RMT-KD for spectral-based model compression, advancing interpretability and robustness in deep learning.

Findings

EigenTrack effectively detects hallucinations in real-time.

RMT-KD produces compact models with minimal accuracy loss.

Spectral statistics serve as robust signals for uncertainty and compression.

Abstract

Large language models and deep neural networks achieve strong performance but suffer from reliability issues and high computational cost. This thesis proposes a unified framework based on spectral geometry and random matrix theory to address both problems by analyzing the eigenvalue structure of hidden activations. The first contribution, EigenTrack, is a real-time method for detecting hallucinations and out-of-distribution behavior in language and vision-language models using spectral features and their temporal dynamics. The second contribution, RMT-KD, is a principled compression method that identifies informative spectral components and applies iterative knowledge distillation to produce compact and efficient models while preserving accuracy. Together, these results show that spectral statistics provide interpretable and robust signals for monitoring uncertainty and guiding compression in large-scale neural networks.