Synthetic Dataset Generation and Validation for Robotic Surgery Instrument Segmentation

TL;DR

This paper introduces a scalable workflow for creating and validating photorealistic synthetic datasets for robotic surgery instrument segmentation, enhancing model training and generalization.

Contribution

It presents an automated pipeline for generating realistic synthetic surgical videos with accurate labels, improving data augmentation and domain adaptation in surgical computer vision.

Findings

Balanced synthetic and real data improves model generalization

Excessive synthetic data causes domain shift

The framework is reproducible and scalable

Abstract

This paper presents a comprehensive workflow for generating and validating a synthetic dataset designed for robotic surgery instrument segmentation. A 3D reconstruction of the Da Vinci robotic arms was refined and animated in Autodesk Maya through a fully automated Python-based pipeline capable of producing photorealistic, labeled video sequences. Each scene integrates randomized motion patterns, lighting variations, and synthetic blood textures to mimic intraoperative variability while preserving pixel-accurate ground truth masks. To validate the realism and effectiveness of the generated data, several segmentation models were trained under controlled ratios of real and synthetic data. Results demonstrate that a balanced composition of real and synthetic samples significantly improves model generalization compared to training on real data only, while excessive reliance on synthetic data introduces a measurable domain shift. The proposed framework provides a reproducible and scalable tool for surgical computer vision, supporting future research in data augmentation, domain adaptation, and simulation-based pretraining for robotic-assisted surgery. Data and code are available at https://github.com/EIDOSLAB/Sintetic-dataset-DaVinci.