MoHETS: Long-term Time Series Forecasting with Mixture-of-Heterogeneous-Experts

TL;DR

MoHETS introduces a novel Transformer-based approach with heterogeneous experts for long-term multivariate time series forecasting, effectively capturing multi-scale structures and improving accuracy over existing methods.

Contribution

The paper proposes MoHETS, a Transformer model with heterogeneous mixture-of-experts layers, enhancing long-term forecasting by capturing diverse temporal dynamics and improving parameter efficiency.

Findings

Achieves state-of-the-art performance on seven benchmarks.

Reduces average MSE by 12% compared to recent baselines.

Effectively models multi-scale and non-stationary time series.

Abstract

Real-world multivariate time series can exhibit intricate multi-scale structures, including global trends, local periodicities, and non-stationary regimes, which makes long-horizon forecasting challenging. Although sparse Mixture-of-Experts (MoE) approaches improve scalability and specialization, they typically rely on homogeneous MLP experts that poorly capture the diverse temporal dynamics of time series data. We address these limitations with MoHETS, an encoder-only Transformer that integrates sparse Mixture-of-Heterogeneous-Experts (MoHE) layers. MoHE routes temporal patches to a small subset of expert networks, combining a shared depthwise-convolution expert for sequence-level continuity with routed Fourier-based experts for patch-level periodic structures. MoHETS further improves robustness to non-stationary dynamics by incorporating exogenous information via cross-attention over covariate patch embeddings. Finally, we replace parameter-heavy linear projection heads with a lightweight convolutional patch decoder, improving parameter efficiency, reducing training instability, and allowing a single model to generalize across arbitrary forecast horizons. We validate across seven multivariate benchmarks and multiple horizons, with MoHETS consistently achieving state-of-the-art performance, reducing the average MSE by $12\%$ compared to strong recent baselines, demonstrating effective heterogeneous specialization for long-term forecasting.