Realistic Surgical Image Dataset Generation Based On 3D Gaussian Splatting

TL;DR

This paper presents a novel method using 3D Gaussian Splatting to generate high-fidelity synthetic surgical datasets, improving training data availability for robotic-assisted minimally invasive surgery and enhancing neural network performance.

Contribution

It introduces a new synthetic data generation technique based on 3D Gaussian Splatting for surgical scenes, enabling better training of vision models in surgery.

Findings

Synthetic data achieved 29.592 PSNR quality.

Models trained on synthetic data outperformed real-data models by 12%.

Synthetic datasets improved neural network performance in surgical scene understanding.

Abstract

Computer vision technologies markedly enhance the automation capabilities of robotic-assisted minimally invasive surgery (RAMIS) through advanced tool tracking, detection, and localization. However, the limited availability of comprehensive surgical datasets for training represents a significant challenge in this field. This research introduces a novel method that employs 3D Gaussian Splatting to generate synthetic surgical datasets. We propose a method for extracting and combining 3D Gaussian representations of surgical instruments and background operating environments, transforming and combining them to generate high-fidelity synthetic surgical scenarios. We developed a data recording system capable of acquiring images alongside tool and camera poses in a surgical scene. Using this pose data, we synthetically replicate the scene, thereby enabling direct comparisons of the synthetic image quality (29.592 PSNR). As a further validation, we compared two YOLOv5 models trained on the synthetic and real data, respectively, and assessed their performance in an unseen real-world test dataset. Comparing the performances, we observe an improvement in neural network performance, with the synthetic-trained model outperforming the real-world trained model by 12%, testing both on real-world data.