Ultrasound Image Generation using Latent Diffusion Models

TL;DR

This paper demonstrates the fine-tuning of latent diffusion models, specifically Stable Diffusion, to generate realistic ultrasound images of the breast, aiding medical training and research with high-quality synthetic data.

Contribution

It introduces a method for fine-tuning large diffusion models on ultrasound datasets to produce realistic medical images with user control via segmentation conditioning.

Findings

Generated images appeared realistic to experts and radiologists.

Successful conditioning with segmentations improved control over generated images.

Source code will be publicly released for community use.

Abstract

Diffusion models for image generation have been a subject of increasing interest due to their ability to generate diverse, high-quality images. Image generation has immense potential in medical imaging because open-source medical images are difficult to obtain compared to natural images, especially for rare conditions. The generated images can be used later to train classification and segmentation models. In this paper, we propose simulating realistic ultrasound (US) images by successive fine-tuning of large diffusion models on different publicly available databases. To do so, we fine-tuned Stable Diffusion, a state-of-the-art latent diffusion model, on BUSI (Breast US Images) an ultrasound breast image dataset. We successfully generated high-quality US images of the breast using simple prompts that specify the organ and pathology, which appeared realistic to three experienced US scientists and a US radiologist. Additionally, we provided user control by conditioning the model with segmentations through ControlNet. We will release the source code at http://code.sonography.ai/ to allow fast US image generation to the scientific community.