SimGen: A Diffusion-Based Framework for Simultaneous Surgical Image and Segmentation Mask Generation

TL;DR

SimGen introduces a diffusion-based framework that simultaneously generates high-quality surgical images and their segmentation masks, reducing reliance on manual annotations for surgical AI applications.

Contribution

The paper presents a novel diffusion model, SimGen, capable of jointly generating surgical images and masks, incorporating cross-correlation priors and a CFL for improved quality and class separation.

Findings

Outperforms baselines in image and mask quality metrics

CFL enhances mask class separability and spatial uniformity

Generated data is suitable for downstream surgical AI tasks

Abstract

Acquiring and annotating surgical data is often resource-intensive, ethical constraining, and requiring significant expert involvement. While generative AI models like text-to-image can alleviate data scarcity, incorporating spatial annotations, such as segmentation masks, is crucial for precision-driven surgical applications, simulation, and education. This study introduces both a novel task and method, SimGen, for Simultaneous Image and Mask Generation. SimGen is a diffusion model based on the DDPM framework and Residual U-Net, designed to jointly generate high-fidelity surgical images and their corresponding segmentation masks. The model leverages cross-correlation priors to capture dependencies between continuous image and discrete mask distributions. Additionally, a Canonical Fibonacci Lattice (CFL) is employed to enhance class separability and uniformity in the RGB space of the masks. SimGen delivers high-fidelity images and accurate segmentation masks, outperforming baselines across six public datasets assessed on image and semantic inception distance metrics. Ablation study shows that the CFL improves mask quality and spatial separation. Downstream experiments suggest generated image-mask pairs are usable if regulations limit human data release for research. This work offers a cost-effective solution for generating paired surgical images and complex labels, advancing surgical AI development by reducing the need for expensive manual annotations.