Perceiver-based CDF Modeling for Time Series Forecasting

TL;DR

This paper introduces perceiver-CDF, a novel model combining perceiver architecture with copula-based attention for efficient, multimodal time series forecasting, achieving significant accuracy improvements with reduced computational costs.

Contribution

The paper presents perceiver-CDF, a new architecture that efficiently models joint distributions of multimodal time series data using perceiver and copula-based attention mechanisms.

Findings

Achieves 20% better accuracy than state-of-the-art methods.

Uses less than half the computational resources.

Effective in both unimodal and multimodal benchmarks.

Abstract

Transformers have demonstrated remarkable efficacy in forecasting time series data. However, their extensive dependence on self-attention mechanisms demands significant computational resources, thereby limiting their practical applicability across diverse tasks, especially in multimodal problems. In this work, we propose a new architecture, called perceiver-CDF, for modeling cumulative distribution functions (CDF) of time series data. Our approach combines the perceiver architecture with a copula-based attention mechanism tailored for multimodal time series prediction. By leveraging the perceiver, our model efficiently transforms high-dimensional and multimodal data into a compact latent space, thereby significantly reducing computational demands. Subsequently, we implement a copula-based attention mechanism to construct the joint distribution of missing data for prediction. Further, we propose an output variance testing mechanism to effectively mitigate error propagation during prediction. To enhance efficiency and reduce complexity, we introduce midpoint inference for the local attention mechanism. This enables the model to efficiently capture dependencies within nearby imputed samples without considering all previous samples. The experiments on the unimodal and multimodal benchmarks consistently demonstrate a 20% improvement over state-of-the-art methods while utilizing less than half of the computational resources.