A Mamba Foundation Model for Time Series Forecasting

TL;DR

This paper introduces TSMamba, a linear-complexity time series foundation model that leverages transfer learning and a novel architecture to achieve high accuracy with less data and computational efficiency.

Contribution

The paper presents TSMamba, a new time series foundation model with linear complexity built on the Mamba architecture, enabling effective zero-shot and few-shot forecasting.

Findings

Zero-shot performance comparable to state-of-the-art models

Achieves high accuracy with less training data

Outperforms task-specific models in full-shot settings

Abstract

Time series foundation models have demonstrated strong performance in zero-shot learning, making them well-suited for predicting rapidly evolving patterns in real-world applications where relevant training data are scarce. However, most of these models rely on the Transformer architecture, which incurs quadratic complexity as input length increases. To address this, we introduce TSMamba, a linear-complexity foundation model for time series forecasting built on the Mamba architecture. The model captures temporal dependencies through both forward and backward Mamba encoders, achieving high prediction accuracy. To reduce reliance on large datasets and lower training costs, TSMamba employs a two-stage transfer learning process that leverages pretrained Mamba LLMs, allowing effective time series modeling with a moderate training set. In the first stage, the forward and backward backbones are optimized via patch-wise autoregressive prediction; in the second stage, the model trains a prediction head and refines other components for long-term forecasting. While the backbone assumes channel independence to manage varying channel numbers across datasets, a channel-wise compressed attention module is introduced to capture cross-channel dependencies during fine-tuning on specific multivariate datasets. Experiments show that TSMamba's zero-shot performance is comparable to state-of-the-art time series foundation models, despite using significantly less training data. It also achieves competitive or superior full-shot performance compared to task-specific prediction models. The code will be made publicly available.