NSTR: Neural Spectral Transport Representation for Space-Varying Frequency Fields

TL;DR

NSTR introduces a novel INR framework that explicitly models space-varying local frequency fields using a learnable PDE, improving signal reconstruction accuracy and interpretability across various data types.

Contribution

It is the first INR method to explicitly model spatially varying spectral characteristics via a learnable frequency transport PDE, enhancing adaptivity and interpretability.

Findings

NSTR outperforms existing INR methods in accuracy-parameter trade-offs.

Requires fewer global frequencies and converges faster.

Provides interpretable visualizations of frequency flows.

Abstract

Implicit Neural Representations (INRs) have emerged as a powerful paradigm for representing signals such as images, audio, and 3D scenes. However, existing INR frameworks -- including MLPs with Fourier features, SIREN, and multiresolution hash grids -- implicitly assume a \textit{global and stationary} spectral basis. This assumption is fundamentally misaligned with real-world signals whose frequency characteristics vary significantly across space, exhibiting local high-frequency textures, smooth regions, and frequency drift phenomena. We propose \textbf{Neural Spectral Transport Representation (NSTR)}, the first INR framework that \textbf{explicitly models a spatially varying local frequency field}. NSTR introduces a learnable \emph{frequency transport equation}, a PDE that governs how local spectral compositions evolve across space. Given a learnable local spectrum field $S(x)$ and a frequency transport network $F_\theta$ enforcing $\nabla S(x) \approx F_\theta(x, S(x))$, NSTR reconstructs signals by spatially modulating a compact set of global sinusoidal bases. This formulation enables strong local adaptivity and offers a new level of interpretability via visualizing frequency flows. Experiments on 2D image regression, audio reconstruction, and implicit 3D geometry show that NSTR achieves significantly better accuracy-parameter trade-offs than SIREN, Fourier-feature MLPs, and Instant-NGP. NSTR requires fewer global frequencies, converges faster, and naturally explains signal structure through spectral transport fields. We believe NSTR opens a new direction in INR research by introducing explicit modeling of space-varying spectrum.