SimuScope: Realistic Endoscopic Synthetic Dataset Generation through Surgical Simulation and Diffusion Models

TL;DR

SimuScope presents a multi-stage pipeline combining surgical simulation and diffusion-based image translation to generate highly realistic synthetic endoscopic data, improving training and guidance for computer-assisted surgery.

Contribution

This work introduces a novel multi-stage pipeline that integrates a surgical simulator with diffusion models to produce realistic annotated synthetic data for CAS systems.

Findings

Generated synthetic data surpasses public datasets in realism and annotation richness.

The diffusion-based translation effectively bridges the gap between synthetic and real images.

The pipeline improves training and guidance in CAS applications.

Abstract

Computer-assisted surgical (CAS) systems enhance surgical execution and outcomes by providing advanced support to surgeons. These systems often rely on deep learning models trained on complex, challenging-to-annotate data. While synthetic data generation can address these challenges, enhancing the realism of such data is crucial. This work introduces a multi-stage pipeline for generating realistic synthetic data, featuring a fully-fledged surgical simulator that automatically produces all necessary annotations for modern CAS systems. This simulator generates a wide set of annotations that surpass those available in public synthetic datasets. Additionally, it offers a more complex and realistic simulation of surgical interactions, including the dynamics between surgical instruments and deformable anatomical environments, outperforming existing approaches. To further bridge the visual gap between synthetic and real data, we propose a lightweight and flexible image-to-image translation method based on Stable Diffusion (SD) and Low-Rank Adaptation (LoRA). This method leverages a limited amount of annotated data, enables efficient training, and maintains the integrity of annotations generated by our simulator. The proposed pipeline is experimentally validated and can translate synthetic images into images with real-world characteristics, which can generalize to real-world context, thereby improving both training and CAS guidance. The code and the dataset are available at https://github.com/SanoScience/SimuScope.