Incremental Transformer Neural Processes

TL;DR

This paper introduces Incremental TNP (incTNP), a novel Transformer Neural Process variant that enables efficient, linear-time updates for streaming data while maintaining high predictive performance and Bayesian consistency.

Contribution

The paper proposes incTNP, which uses causal masking and caching to achieve fast, incremental updates in Transformer Neural Processes without sacrificing accuracy or Bayesian properties.

Findings

incTNP matches or exceeds standard TNP performance.

incTNP reduces update complexity from quadratic to linear.

incTNP maintains Bayesian consistency in streaming inference.

Abstract

Neural Processes (NPs), and specifically Transformer Neural Processes (TNPs), have demonstrated remarkable performance across tasks ranging from spatiotemporal forecasting to tabular data modelling. However, many of these applications are inherently sequential, involving continuous data streams such as real-time sensor readings or database updates. In such settings, models should support cheap, incremental updates rather than recomputing internal representations from scratch for every new observation -- a capability existing TNP variants lack. Drawing inspiration from Large Language Models, we introduce the Incremental TNP (incTNP). By leveraging causal masking, Key-Value (KV) caching, and a data-efficient autoregressive training strategy, incTNP matches the predictive performance of standard TNPs while reducing the computational cost of updates from quadratic to linear time complexity. We empirically evaluate our model on a range of synthetic and real-world tasks, including tabular regression and temperature prediction. Our results show that, surprisingly, incTNP delivers performance comparable to -- or better than -- non-causal TNPs while unlocking orders-of-magnitude speedups for sequential inference. Finally, we assess the consistency of the model's updates -- by adapting a metric of ``implicit Bayesianness", we show that incTNP retains a prediction rule as implicitly Bayesian as standard non-causal TNPs, demonstrating that incTNP achieves the computational benefits of causal masking without sacrificing the consistency required for streaming inference.