Med-Art: Diffusion Transformer for 2D Medical Text-to-Image Generation

TL;DR

Med-Art introduces a diffusion transformer framework tailored for medical text-to-image generation, effectively addressing data scarcity and improving image quality through innovative fine-tuning methods.

Contribution

The paper presents Med-Art, a novel framework that adapts large-scale pre-trained models with hybrid-level diffusion fine-tuning for medical image synthesis with limited data.

Findings

Achieves state-of-the-art FID and KID scores on medical datasets.

Effectively handles small datasets and textual data scarcity.

Improves image quality and downstream classification performance.

Abstract

Text-to-image generative models have achieved remarkable breakthroughs in recent years. However, their application in medical image generation still faces significant challenges, including small dataset sizes, and scarcity of medical textual data. To address these challenges, we propose Med-Art, a framework specifically designed for medical image generation with limited data. Med-Art leverages vision-language models to generate visual descriptions of medical images which overcomes the scarcity of applicable medical textual data. Med-Art adapts a large-scale pre-trained text-to-image model, PixArt-$\alpha$, based on the Diffusion Transformer (DiT), achieving high performance under limited data. Furthermore, we propose an innovative Hybrid-Level Diffusion Fine-tuning (HLDF) method, which enables pixel-level losses, effectively addressing issues such as overly saturated colors. We achieve state-of-the-art performance on two medical image datasets, measured by FID, KID, and downstream classification performance.