Diffusion Model in Latent Space for Medical Image Segmentation Task

TL;DR

MedSegLatDiff introduces a diffusion-based framework in latent space combining VAE and diffusion models for efficient, diverse, and interpretable medical image segmentation, outperforming existing methods on multiple datasets.

Contribution

The paper presents a novel latent diffusion framework with a VAE for medical segmentation, enabling efficient, diverse, and reliable predictions with improved structural preservation.

Findings

Achieves state-of-the-art Dice and IoU scores on multiple datasets.

Generates diverse segmentation hypotheses and confidence maps.

Enhances interpretability and reliability over deterministic methods.

Abstract

Medical image segmentation is crucial for clinical diagnosis and treatment planning. Traditional methods typically produce a single segmentation mask, failing to capture inherent uncertainty. Recent generative models enable the creation of multiple plausible masks per image, mimicking the collaborative interpretation of several clinicians. However, these approaches remain computationally heavy. We propose MedSegLatDiff, a diffusion based framework that combines a variational autoencoder (VAE) with a latent diffusion model for efficient medical image segmentation. The VAE compresses the input into a low dimensional latent space, reducing noise and accelerating training, while the diffusion process operates directly in this compact representation. We further replace the conventional MSE loss with weighted cross entropy in the VAE mask reconstruction path to better preserve tiny structures such as small nodules. MedSegLatDiff is evaluated on ISIC-2018 (skin lesions), CVC-Clinic (polyps), and LIDC-IDRI (lung nodules). It achieves state of the art or highly competitive Dice and IoU scores while simultaneously generating diverse segmentation hypotheses and confidence maps. This provides enhanced interpretability and reliability compared to deterministic baselines, making the model particularly suitable for clinical deployment.