ChunkFormer: Learning Long Time Series with Multi-stage Chunked Transformer

TL;DR

ChunkFormer is a novel Transformer-based architecture designed to efficiently analyze long time series data by progressively chunking sequences, capturing both local and global information while reducing computational resources.

Contribution

The paper introduces ChunkFormer, a multi-stage chunking approach that enhances Transformer models for long sequence analysis without increasing resource consumption.

Findings

Outperforms existing Transformer models on various long sequence tasks.

Effectively captures local seasonality and fluctuations in long time series.

Reduces computational resources needed for training long sequences.

Abstract

The analysis of long sequence data remains challenging in many real-world applications. We propose a novel architecture, ChunkFormer, that improves the existing Transformer framework to handle the challenges while dealing with long time series. Original Transformer-based models adopt an attention mechanism to discover global information along a sequence to leverage the contextual data. Long sequential data traps local information such as seasonality and fluctuations in short data sequences. In addition, the original Transformer consumes more resources by carrying the entire attention matrix during the training course. To overcome these challenges, ChunkFormer splits the long sequences into smaller sequence chunks for the attention calculation, progressively applying different chunk sizes in each stage. In this way, the proposed model gradually learns both local and global information without changing the total length of the input sequences. We have extensively tested the effectiveness of this new architecture on different business domains and have proved the advantage of such a model over the existing Transformer-based models.