Anchoring the Eigengap: Cross-Modal Spectral Stabilization for Sample-Efficient Representation Learning

TL;DR

This paper introduces a spectral theory framework for understanding and improving low-data representation learning, especially in medical imaging, by stabilizing eigengaps through multimodal learning and spectral filtering.

Contribution

It develops a theory linking eigenvalue decay to data efficiency, proposing spectral stabilization via multimodal learning to enhance low-data model performance.

Findings

Multimodal learning preserves more stable spectral modes in low-data regimes.

Spectral collapse limits the number of recoverable signal modes, affecting classification.

Zeta-based spectral filtering improves data efficiency and model stability.

Abstract

Deep vision models degrade sharply in low-data regimes, particularly in medical imaging where labeled samples are scarce. We show this arises not merely from overfitting but from a geometric failure: finite-sample noise corrupts the embedding covariance, collapsing the eigengap and limiting the number of recoverable signal-bearing modes. We develop a spectral theory of finite-sample representation learning that quantifies the recoverable dimension K(N), the number of eigenmodes that can be stably estimated from N samples. Using perturbation theory and concentration bounds, we show that only modes with eigenvalues above the noise floor $\|\hat{\Sigma} - \Sigma\|_{\mathrm{op}} \sim \sqrt{D/N}$ are reliable, yielding a truncated Mahalanobis energy that governs classification performance. Under a power-law spectral model, this energy can be approximated by a truncated Riemann zeta function, linking eigenvalue decay to data efficiency and AUC. Within this framework, multimodal learning acts as spectral stabilization: vision-language models impose low-rank constraints that suppress noise-dominated directions and preserve the eigengap, increasing K(N) under data scarcity. Across MNIST and multi-disease neuroimaging, we show that multimodal training maintains more stable modes and improves class separation, even when unimodal models achieve comparable few-shot accuracy. These results identify spectral collapse as a fundamental bottleneck in low-data learning. We use truncated Mahalanobis energy and K(N) to diagnose encoder quality, and introduce zeta-based spectral filtering as a principled approach to improve data efficiency.