FMamba: Mamba based on Fast-attention for Multivariate Time-series Forecasting

TL;DR

FMamba is a novel multivariate time-series forecasting model that combines fast-attention with Mamba, achieving state-of-the-art accuracy with low computational cost by effectively capturing temporal and inter-variable dependencies.

Contribution

This paper introduces FMamba, integrating fast-attention with Mamba to model variable relationships and temporal features efficiently in multivariate time-series forecasting.

Findings

FMamba outperforms existing models on eight public datasets.

FMamba maintains low computational overhead while achieving high accuracy.

The model effectively captures both temporal and inter-variable dependencies.

Abstract

In multivariate time-series forecasting (MTSF), extracting the temporal correlations of the input sequences is crucial. While popular Transformer-based predictive models can perform well, their quadratic computational complexity results in inefficiency and high overhead. The recently emerged Mamba, a selective state space model, has shown promising results in many fields due to its strong temporal feature extraction capabilities and linear computational complexity. However, due to the unilateral nature of Mamba, channel-independent predictive models based on Mamba cannot attend to the relationships among all variables in the manner of Transformer-based models. To address this issue, we combine fast-attention with Mamba to introduce a novel framework named FMamba for MTSF. Technically, we first extract the temporal features of the input variables through an embedding layer, then compute the dependencies among input variables via the fast-attention module. Subsequently, we use Mamba to selectively deal with the input features and further extract the temporal dependencies of the variables through the multi-layer perceptron block (MLP-block). Finally, FMamba obtains the predictive results through the projector, a linear layer. Experimental results on eight public datasets demonstrate that FMamba can achieve state-of-the-art performance while maintaining low computational overhead.