Whole Heart 3D+T Representation Learning Through Sparse 2D Cardiac MR Images

TL;DR

This paper introduces a self-supervised learning framework for 3D+T cardiac MRI data that automatically learns comprehensive heart representations from unlabeled images, improving downstream cardiac analysis tasks.

Contribution

The work presents a novel masked imaging modeling approach for self-supervised learning of 3D+T cardiac representations from unlabeled data, enhancing robustness and downstream task performance.

Findings

Outperforms baseline methods in cardiac phenotype prediction

Achieves superior results in multi-plane CMR segmentation

Demonstrates robustness to missing or flawed CMR planes

Abstract

Cardiac Magnetic Resonance (CMR) imaging serves as the gold-standard for evaluating cardiac morphology and function. Typically, a multi-view CMR stack, covering short-axis (SA) and 2/3/4-chamber long-axis (LA) views, is acquired for a thorough cardiac assessment. However, efficiently streamlining the complex, high-dimensional 3D+T CMR data and distilling compact, coherent representation remains a challenge. In this work, we introduce a whole-heart self-supervised learning framework that utilizes masked imaging modeling to automatically uncover the correlations between spatial and temporal patches throughout the cardiac stacks. This process facilitates the generation of meaningful and well-clustered heart representations without relying on the traditionally required, and often costly, labeled data. The learned heart representation can be directly used for various downstream tasks. Furthermore, our method demonstrates remarkable robustness, ensuring consistent representations even when certain CMR planes are missing/flawed. We train our model on 14,000 unlabeled CMR data from UK BioBank and evaluate it on 1,000 annotated data. The proposed method demonstrates superior performance to baselines in tasks that demand comprehensive 3D+T cardiac information, e.g. cardiac phenotype (ejection fraction and ventricle volume) prediction and multi-plane/multi-frame CMR segmentation, highlighting its effectiveness in extracting comprehensive cardiac features that are both anatomically and pathologically relevant.