Taylorformer: Probabilistic Modelling for Random Processes including Time Series

TL;DR

Taylorformer introduces a novel probabilistic model for random processes like time series, combining Taylor approximations and a specialized attention mechanism to improve predictive accuracy and uncertainty estimation.

Contribution

It presents the Taylorformer, integrating LocalTaylor and MHA-X components, achieving state-of-the-art results in neural process tasks and forecasting accuracy.

Findings

Outperforms state-of-the-art in neural process log-likelihoods

Achieves at least 14% MSE improvement in forecasting tasks

Provides uncertainty-aware predictions for stochastic processes

Abstract

We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14\% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material.