Mamba or Transformer for Time Series Forecasting? Mixture of Universals (MoU) Is All You Need

TL;DR

The paper introduces MoU, a versatile model combining multiple architectures to effectively capture both short-term and long-term dependencies in time series forecasting, achieving state-of-the-art results with low computational cost.

Contribution

MoU is a novel hybrid model that integrates feature extractors and multiple architectures to improve long-term and short-term dependency modeling in time series forecasting.

Findings

MoU outperforms existing models on seven real-world datasets.

MoU maintains low computational costs while achieving high accuracy.

The hybrid architecture effectively captures diverse dependency types.

Abstract

Time series forecasting requires balancing short-term and long-term dependencies for accurate predictions. Existing methods mainly focus on long-term dependency modeling, neglecting the complexities of short-term dynamics, which may hinder performance. Transformers are superior in modeling long-term dependencies but are criticized for their quadratic computational cost. Mamba provides a near-linear alternative but is reported less effective in time series longterm forecasting due to potential information loss. Current architectures fall short in offering both high efficiency and strong performance for long-term dependency modeling. To address these challenges, we introduce Mixture of Universals (MoU), a versatile model to capture both short-term and long-term dependencies for enhancing performance in time series forecasting. MoU is composed of two novel designs: Mixture of Feature Extractors (MoF), an adaptive method designed to improve time series patch representations for short-term dependency, and Mixture of Architectures (MoA), which hierarchically integrates Mamba, FeedForward, Convolution, and Self-Attention architectures in a specialized order to model long-term dependency from a hybrid perspective. The proposed approach achieves state-of-the-art performance while maintaining relatively low computational costs. Extensive experiments on seven real-world datasets demonstrate the superiority of MoU. Code is available at https://github.com/lunaaa95/mou/.