MSDformer: Multi-scale Discrete Transformer For Time Series Generation

TL;DR

MSDformer introduces a multi-scale discrete transformer approach for time series generation, effectively capturing complex patterns and providing theoretical validation, leading to superior performance over existing methods.

Contribution

The paper presents a novel multi-scale DTM-based method, MSDformer, with a multi-scale tokenizer and modeling technique, supported by theoretical analysis and improved experimental results.

Findings

MSDformer outperforms state-of-the-art methods in time series generation.

Theoretical validation via rate-distortion theorem supports the approach.

Multi-scale modeling enhances the quality of generated time series.

Abstract

Discrete Token Modeling (DTM), which employs vector quantization techniques, has demonstrated remarkable success in modeling non-natural language modalities, particularly in time series generation. While our prior work SDformer established the first DTM-based framework to achieve state-of-the-art performance in this domain, two critical limitations persist in existing DTM approaches: 1) their inability to capture multi-scale temporal patterns inherent to complex time series data, and 2) the absence of theoretical foundations to guide model optimization. To address these challenges, we proposes a novel multi-scale DTM-based time series generation method, called Multi-Scale Discrete Transformer (MSDformer). MSDformer employs a multi-scale time series tokenizer to learn discrete token representations at multiple scales, which jointly characterize the complex nature of time series data. Subsequently, MSDformer applies a multi-scale autoregressive token modeling technique to capture the multi-scale patterns of time series within the discrete latent space. Theoretically, we validate the effectiveness of the DTM method and the rationality of MSDformer through the rate-distortion theorem. Comprehensive experiments demonstrate that MSDformer significantly outperforms state-of-the-art methods. Both theoretical analysis and experimental results demonstrate that incorporating multi-scale information and modeling multi-scale patterns can substantially enhance the quality of generated time series in DTM-based approaches. Code is available at this repository:https://github.com/kkking-kk/MSDformer.