Spectral Transformer Neural Processes

TL;DR

Spectral Transformer Neural Processes (STNPs) enhance Neural Processes by integrating spectral features, significantly improving modeling of periodic and quasi-periodic data across various applications.

Contribution

The paper introduces STNPs, a spectral-aware extension of Transformer Neural Processes, which effectively model periodicity by incorporating spectral mixture features.

Findings

STNPs outperform existing methods on synthetic and real-world datasets.

The spectral mixture approach improves modeling of periodic and quasi-periodic data.

Experiments demonstrate enhanced predictive accuracy across multiple domains.

Abstract

Time series, spatial data, and images are natural applications of Neural Processes. However, when such data exhibit strong periodicity and quasi-periodicity, existing methods often suffer from underfitting and generalise poorly beyond the training distribution. In this work, we propose Spectral Transformer Neural Processes (STNPs), a frequency-aware extension of Transformer Neural Processes (TNPs). STNPs introduce a Spectral Aggregator that estimates an empirical context spectrum, compresses it into a spectral mixture, samples task-adaptive spectral features, and concatenates them with time-domain embeddings, thereby injecting a spectral-mixture-kernel bias into TNPs. This design reshapes the similarity geometry, allowing inputs that are distant in Euclidean space to remain close in an induced periodic manifold while enhancing time-frequency interactions. Extensive experiments on synthetic regression tasks, real-world time-series datasets, and an image dataset demonstrate that STNPs consistently improve predictive performance over existing baselines, extending Neural Processes beyond translation equivariance towards effective modelling of periodicity and quasi-periodicity.