Flexible Conditional Image Generation of Missing Data with Learned Mental Maps

TL;DR

This paper introduces a novel method leveraging Bayesian Deep Learning and environment mapping to generate full 3D anatomical images from sparse 2D slices, aiding diagnostics when high-resolution data is unavailable.

Contribution

It presents a new approach combining GQN and Conditional BRUNO for volumetric reconstruction from limited slices, applicable to clinical imaging scenarios.

Findings

Achieves SSIM of 0.7+ for 3D reconstructions from 1-4 slices.

Attains cross-correlation of 0.8+ with ground truth.

Demonstrates effectiveness on abdominal CT, brain MRI, and fetal MRI data.

Abstract

Real-world settings often do not allow acquisition of high-resolution volumetric images for accurate morphological assessment and diagnostic. In clinical practice it is frequently common to acquire only sparse data (e.g. individual slices) for initial diagnostic decision making. Thereby, physicians rely on their prior knowledge (or mental maps) of the human anatomy to extrapolate the underlying 3D information. Accurate mental maps require years of anatomy training, which in the first instance relies on normative learning, i.e. excluding pathology. In this paper, we leverage Bayesian Deep Learning and environment mapping to generate full volumetric anatomy representations from none to a small, sparse set of slices. We evaluate proof of concept implementations based on Generative Query Networks (GQN) and Conditional BRUNO using abdominal CT and brain MRI as well as in a clinical application involving sparse, motion-corrupted MR acquisition for fetal imaging. Our approach allows to reconstruct 3D volumes from 1 to 4 tomographic slices, with a SSIM of 0.7+ and cross-correlation of 0.8+ compared to the 3D ground truth.