Decentralized Attention Fails Centralized Signals: Rethinking Transformers for Medical Time Series

TL;DR

This paper introduces CoTAR, a centralized module replacing traditional decentralized attention in Transformer models, to better capture global patterns in medical time series data, improving accuracy and efficiency.

Contribution

Proposes CoTAR, a centralized MLP-based module that aligns with the inherent centralization of MedTS signals, reducing complexity and enhancing performance over standard attention mechanisms.

Findings

Achieves up to 11.6% improvement on the APAVA dataset.

Reduces memory usage to 33% and inference time to 20% of previous methods.

Demonstrates superior effectiveness and efficiency on five benchmark datasets.

Abstract

Accurate analysis of medical time series (MedTS) data, such as electroencephalography (EEG) and electrocardiography (ECG), plays a pivotal role in healthcare applications, including the diagnosis of brain and heart diseases. MedTS data typically exhibit two critical patterns: temporal dependencies within individual channels and channel dependencies across multiple channels. While recent advances in deep learning have leveraged Transformer-based models to effectively capture temporal dependencies, they often struggle with modeling channel dependencies. This limitation stems from a structural mismatch: MedTS signals are inherently centralized, whereas the Transformer's attention mechanism is decentralized, making it less effective at capturing global synchronization and unified waveform patterns. To address this mismatch, we propose CoTAR (Core Token Aggregation-Redistribution), a centralized MLP-based module designed to replace decentralized attention. Instead of allowing all tokens to interact directly, as in standard attention, CoTAR introduces a global core token that serves as a proxy to facilitate inter-token interactions, thereby enforcing a centralized aggregation and redistribution strategy. This design not only better aligns with the centralized nature of MedTS signals but also reduces computational complexity from quadratic to linear. Experiments on five benchmarks validate the superiority of our method in both effectiveness and efficiency, achieving up to a 11.6% improvement on the APAVA dataset, while using only 33% of the memory and 20% of the inference time compared to the previous state of the art. Code and all training scripts are available at https://github.com/Levi-Ackman/TeCh.