Surgical Text-to-Image Generation

TL;DR

This paper adapts text-to-image generative models for surgical images using triplet-based captions, achieving high-quality, realistic images that could reduce reliance on costly real data collection.

Contribution

It introduces Surgical Imagen, a diffusion-based model tailored for surgical data, with novel triplet-based caption handling and class balancing techniques for improved image synthesis.

Findings

Achieved FID score of 3.7 indicating high image quality.

Human experts rated generated images as highly realistic.

Model effectively aligns images with complex surgical action descriptions.

Abstract

Acquiring surgical data for research and development is significantly hindered by high annotation costs and practical and ethical constraints. Utilizing synthetically generated images could offer a valuable alternative. In this work, we explore adapting text-to-image generative models for the surgical domain using the CholecT50 dataset, which provides surgical images annotated with action triplets (instrument, verb, target). We investigate several language models and find T5 to offer more distinct features for differentiating surgical actions on triplet-based textual inputs, and showcasing stronger alignment between long and triplet-based captions. To address challenges in training text-to-image models solely on triplet-based captions without additional inputs and supervisory signals, we discover that triplet text embeddings are instrument-centric in the latent space. Leveraging this insight, we design an instrument-based class balancing technique to counteract data imbalance and skewness, improving training convergence. Extending Imagen, a diffusion-based generative model, we develop Surgical Imagen to generate photorealistic and activity-aligned surgical images from triplet-based textual prompts. We assess the model on quality, alignment, reasoning, and knowledge, achieving FID and CLIP scores of 3.7 and 26.8% respectively. Human expert survey shows that participants were highly challenged by the realistic characteristics of the generated samples, demonstrating Surgical Imagen's effectiveness as a practical alternative to real data collection.