Integrating Sequence and Image Modeling in Irregular Medical Time Series Through Self-Supervised Learning

TL;DR

This paper introduces a joint learning framework that combines sequence and image modeling with self-supervised strategies to improve analysis of irregular, missing medical time series data, outperforming existing models.

Contribution

It presents a novel integrated approach with self-supervised fusion strategies for sequence and image representations in medical time series analysis.

Findings

Outperforms seven state-of-the-art models on clinical datasets.

More robust to missing data simulated by leave-sensors-out and leave-samples-out.

Achieves significant classification accuracy improvements.

Abstract

Medical time series are often irregular and face significant missingness, posing challenges for data analysis and clinical decision-making. Existing methods typically adopt a single modeling perspective, either treating series data as sequences or transforming them into image representations for further classification. In this paper, we propose a joint learning framework that incorporates both sequence and image representations. We also design three self-supervised learning strategies to facilitate the fusion of sequence and image representations, capturing a more generalizable joint representation. The results indicate that our approach outperforms seven other state-of-the-art models in three representative real-world clinical datasets. We further validate our approach by simulating two major types of real-world missingness through leave-sensors-out and leave-samples-out techniques. The results demonstrate that our approach is more robust and significantly surpasses other baselines in terms of classification performance.