Visually-Guided Controllable Medical Image Generation via Fine-Grained Semantic Disentanglement

TL;DR

This paper introduces a novel framework for medical image generation that uses visual priors and semantic disentanglement to improve controllability and quality in text-to-image synthesis, addressing modality gaps and semantic entanglement issues.

Contribution

We propose a cross-modal latent alignment and hybrid feature fusion approach that enhances fine-grained control and generation quality in medical image synthesis.

Findings

Outperforms existing methods in generation quality

Improves downstream classification performance

Demonstrates effectiveness across three datasets

Abstract

Medical image synthesis is crucial for alleviating data scarcity and privacy constraints. However, fine-tuning general text-to-image (T2I) models remains challenging, mainly due to the significant modality gap between complex visual details and abstract clinical text. In addition, semantic entanglement persists, where coarse-grained text embeddings blur the boundary between anatomical structures and imaging styles, thus weakening controllability during generation. To address this, we propose a Visually-Guided Text Disentanglement framework. We introduce a cross-modal latent alignment mechanism that leverages visual priors to explicitly disentangle unstructured text into independent semantic representations. Subsequently, a Hybrid Feature Fusion Module (HFFM) injects these features into a Diffusion Transformer (DiT) through separated channels, enabling fine-grained structural control. Experimental results in three datasets demonstrate that our method outperforms existing approaches in terms of generation quality and significantly improves performance on downstream classification tasks. The source code is available at https://github.com/hx111/VG-MedGen.