SDE-Attention: Latent Attention in SDE-RNNs for Irregularly Sampled Time Series with Missing Data

TL;DR

SDE-Attention introduces channel-level attention mechanisms in SDE-RNNs to effectively handle irregularly sampled time series with missing data, improving accuracy across synthetic and real-world benchmarks.

Contribution

The paper proposes SDE-Attention, a novel family of SDE-RNNs with attention modules for better modeling of irregular time series with missing data, demonstrating significant performance gains.

Findings

Latent-space attention improves SDE-RNN performance over vanilla models.

SDE-TVF-L achieves up to 10% accuracy improvement on univariate datasets.

Attention mechanisms enhance multivariate time series modeling, with different types excelling on different tasks.

Abstract

Irregularly sampled time series with substantial missing observations are common in healthcare and sensor networks. We introduce SDE-Attention, a family of SDE-RNNs equipped with channel-level attention on the latent pre-RNN state, including channel recalibration, time-varying feature attention, and pyramidal multi-scale self-attention. We therefore conduct a comparison on a synthetic periodic dataset and real-world benchmarks, under varying missing rate. Latent-space attention consistently improves over a vanilla SDE-RNN. On the univariate UCR datasets, the LSTM-based time-varying feature model SDE-TVF-L achieves the highest average accuracy, raising mean performance by approximately 4, 6, and 10 percentage points over the baseline at 30%, 60% and 90% missingness, respectively (averaged across datasets). On multivariate UEA benchmarks, attention-augmented models again outperform the backbone, with SDE-TVF-L yielding up to a 7% gain in mean accuracy under high missingness. Among the proposed mechanisms, time-varying feature attention is the most robust on univariate datasets. On multivariate datasets, different attention types excel on different tasks, showing that SDE-Attention can be flexibly adapted to the structure of each problem.