Sketch2CT: Multimodal Diffusion for Structure-Aware 3D Medical Volume Generation

TL;DR

Sketch2CT introduces a multimodal diffusion framework that generates anatomically consistent 3D medical volumes guided by user sketches and text, improving data augmentation and realism in medical imaging.

Contribution

It presents a novel structure-aware 3D medical volume generation method combining sketch and text guidance with a capsule-attention backbone.

Findings

Outperforms existing methods on public CT datasets

Enables controllable and efficient 3D volume synthesis

Produces anatomically accurate and realistic medical images

Abstract

Diffusion probabilistic models have demonstrated significant potential in generating high-quality, realistic medical images, providing a promising solution to the persistent challenge of data scarcity in the medical field. Nevertheless, producing 3D medical volumes with anatomically consistent structures under multimodal conditions remains a complex and unresolved problem. We introduce Sketch2CT, a multimodal diffusion framework for structure-aware 3D medical volume generation, jointly guided by a user-provided 2D sketch and a textual description that captures 3D geometric semantics. The framework initially generates 3D segmentation masks of the target organ from random noise, conditioned on both modalities. To effectively align and fuse these inputs, we propose two key modules that refine sketch features with localized textual cues and integrate global sketch-text representations. Built upon a capsule-attention backbone, these modules leverage the complementary strengths of sketches and text to produce anatomically accurate organ shapes. The synthesized segmentation masks subsequently guide a latent diffusion model for 3D CT volume synthesis, enabling realistic reconstruction of organ appearances that are consistent with user-defined sketches and descriptions. Extensive experiments on public CT datasets demonstrate that Sketch2CT achieves superior performance in generating multimodal medical volumes. Its controllable, low-cost generation pipeline enables principled, efficient augmentation of medical datasets. Code is available at https://github.com/adlsn/Sketch2CT.