MoDEx: Mixture of Depth-specific Experts for Multivariate Long-term Time Series Forecasting

TL;DR

MoDEx introduces a novel mixture of depth-specific experts for multivariate long-term time series forecasting, leveraging layer sensitivity to improve accuracy and efficiency across various benchmarks.

Contribution

The paper proposes MoDEx, a lightweight mixture of depth-specific MLP experts, inspired by layer sensitivity analysis, to enhance long-term time series forecasting performance.

Findings

Achieves state-of-the-art accuracy on seven benchmarks.

Ranks first in 78% of evaluated cases.

Uses fewer parameters and computational resources.

Abstract

Multivariate long-term time series forecasting (LTSF) supports critical applications such as traffic-flow management, solar-power scheduling, and electricity-transformer monitoring. The existing LTSF paradigms follow a three-stage pipeline of embedding, backbone refinement, and long-horizon prediction. However, the behaviors of individual backbone layers remain underexplored. We introduce layer sensitivity, a gradient-based metric inspired by GradCAM and effective receptive field theory, which quantifies both positive and negative contributions of each time point to a layer's latent features. Applying this metric to a three-layer MLP backbone reveals depth-specific specialization in modeling temporal dynamics in the input sequence. Motivated by these insights, we propose MoDEx, a lightweight Mixture of Depth-specific Experts, which replaces complex backbones with depth-specific MLP experts. MoDEx achieves state-of-the-art accuracy on seven real-world benchmarks, ranking first in 78 percent of cases, while using significantly fewer parameters and computational resources. It also integrates seamlessly into transformer variants, consistently boosting their performance and demonstrating robust generalizability as an efficient and high-performance LTSF framework.