Compact Circulant Layers with Spectral Priors

TL;DR

This paper introduces spectral circulant and BCCB neural network layers that are memory-efficient, enable spectral structure imposition, and provide exact spectral norms, improving robustness and efficiency in resource-constrained settings.

Contribution

It proposes a novel spectral parameterization of circulant layers, enabling structured variational inference and exact spectral norm computation for compact neural networks.

Findings

Spectral circulant layers perform well on MNIST and CIFAR-10.

They achieve comparable accuracy with fewer parameters.

They provide tighter Lipschitz bounds and robustness diagnostics.

Abstract

Critical applications in areas such as medicine, robotics and autonomous systems require compact (i.e., memory efficient), uncertainty-aware neural networks suitable for edge and other resource-constrained deployments. We study compact spectral circulant and block-circulant-with-circulant-blocks (BCCB) layers: FFT-diagonalizable circular convolutions whose weights live directly in the real FFT (RFFT) half (1D) or half-plane (2D). Parameterizing filters in the frequency domain lets us impose simple spectral structure, perform structured variational inference in a low-dimensional weight space, and calculate exact layer spectral norms, enabling inexpensive global Lipschitz bounds and margin-based robustness diagnostics. By placing independent complex Gaussians on the Hermitian support we obtain a discrete instance of the spectral representation of stationary kernels, inducing an exact stationary Gaussian-process prior over filters on the discrete circle/torus. We exploit this to define a practical spectral prior and a Hermitian-aware low-rank-plus-diagonal variational posterior in real coordinates. Empirically, spectral circulant/BCCB layers are effective compact building blocks in both (variational) Bayesian and point estimate regimes: compact Bayesian neural networks on MNIST->Fashion-MNIST, variational heads on frozen CIFAR-10 features, and deterministic ViT projections on CIFAR-10/Tiny ImageNet; spectral layers match strong baselines while using substantially fewer parameters and with tighter Lipschitz certificates.