Medical data wrangling with sequential variational autoencoders

TL;DR

This paper introduces Shi-VAE, a novel sequential variational autoencoder model designed to handle heterogeneous medical data with bursty missing patterns, improving data imputation and analysis in healthcare settings.

Contribution

The paper presents Shi-VAE, a new methodology extending VAEs to sequential medical data with bursty missingness, outperforming state-of-the-art models in accuracy and efficiency.

Findings

Shi-VAE achieves superior imputation performance using RMSE and cross-correlation metrics.

Shi-VAE has lower computational complexity compared to GP-VAE.

The model effectively handles heterogeneous data types and bursty missing patterns.

Abstract

Medical data sets are usually corrupted by noise and missing data. These missing patterns are commonly assumed to be completely random, but in medical scenarios, the reality is that these patterns occur in bursts due to sensors that are off for some time or data collected in a misaligned uneven fashion, among other causes. This paper proposes to model medical data records with heterogeneous data types and bursty missing data using sequential variational autoencoders (VAEs). In particular, we propose a new methodology, the Shi-VAE, which extends the capabilities of VAEs to sequential streams of data with missing observations. We compare our model against state-of-the-art solutions in an intensive care unit database (ICU) and a dataset of passive human monitoring. Furthermore, we find that standard error metrics such as RMSE are not conclusive enough to assess temporal models and include in our analysis the cross-correlation between the ground truth and the imputed signal. We show that Shi-VAE achieves the best performance in terms of using both metrics, with lower computational complexity than the GP-VAE model, which is the state-of-the-art method for medical records.