JustDense: Just using Dense instead of Sequence Mixer for Time Series analysis

TL;DR

This paper investigates whether complex sequence mixers are necessary for time series analysis by replacing them with dense layers, finding that simpler architectures often perform equally or better across multiple benchmarks.

Contribution

It introduces JustDense, a systematic approach replacing sequence mixers with dense layers, challenging the necessity of complex mixers in TSA models.

Findings

Replacing sequence mixers with dense layers achieves comparable or superior performance.

Simpler dense architectures can outperform complex mixers in TSA tasks.

Challenges the assumption that more complex models are inherently better for time series analysis.

Abstract

Sequence and channel mixers, the core mechanism in sequence models, have become the de facto standard in time series analysis (TSA). However, recent studies have questioned the necessity of complex sequence mixers, such as attention mechanisms, demonstrating that simpler architectures can achieve comparable or even superior performance. This suggests that the benefits attributed to complex sequencemixers might instead emerge from other architectural or optimization factors. Based on this observation, we pose a central question: Are common sequence mixers necessary for time-series analysis? Therefore, we propose JustDense, an empirical study that systematically replaces sequence mixers in various well-established TSA models with dense layers. Grounded in the MatrixMixer framework, JustDense treats any sequence mixer as a mixing matrix and replaces it with a dense layer. This substitution isolates the mixing operation, enabling a clear theoretical foundation for understanding its role. Therefore, we conducted extensive experiments on 29 benchmarks covering five representative TSA tasks using seven state-of-the-art TSA models to address our research question. The results show that replacing sequence mixers with dense layers yields comparable or even superior performance. In the cases where dedicated sequence mixers still offer benefits, JustDense challenges the assumption that "deeper and more complex architectures are inherently better" in TSA.