Diffusion-Guided Feature Selection via Nishimori Temperature: Noise-Based Spectral Embedding

TL;DR

This paper introduces Noise-Based Spectral Embedding (NBSE), a physics-inspired method for feature selection that maintains high classification accuracy under aggressive data compression.

Contribution

NBSE leverages Nishimori temperature and spectral analysis to identify and select informative features without greedy search, improving robustness and efficiency.

Findings

NBSE preserves classification accuracy with only 30% features on ImageNet embeddings.

It outperforms ANOVA F-test and random selection by up to 6.8%.

The method is robust to measurement noise, with limited shift in Nishimori temperature.

Abstract

We propose Noise-Based Spectral Embedding (NBSE), a physics-informed framework for selecting informative features from high-dimensional data without greedy search. NBSE constructs a sparse similarity graph on the samples and identifies the Nishimori temperature $\beta_N$ the critical inverse temperature at which the Bethe Hessian becomes singular. The corresponding smallest eigenvector captures the dominant mode of an intrinsically degree-corrected diffusion process, naturally reweighting nodes to prevent hub dominance. By transposing the data matrix and applying NBSE in feature space, we obtain a one-dimensional spectral embedding that reveals groups of redundant or semantically related dimensions; balanced binning then selects one representative per group. We prove that coloured Gaussian perturbations shift $\beta_N$ by at most $O(\bar\sigma^2)$, guaranteeing robustness to measurement noise. Experiments on ImageNet embeddings from MobileNetV2 and EfficientNet-B4 show that NBSE preserves classification accuracy even under aggressive compression: on EfficientNet-B4 the accuracy drop is below $1\%$ when retaining only $30\%$ of features, outperforming ANOVA $F$-test and random selection by up to $6.8\%$.