Generalizing Surgical Instruments Segmentation to Unseen Domains with One-to-Many Synthesis

TL;DR

This paper presents a novel method for generating synthetic surgical instrument datasets from minimal source images to improve deep learning model generalization across unseen real-world domains.

Contribution

The authors introduce a one-to-many synthetic data generation framework using limited source images and blending techniques to enhance surgical instrument segmentation in unseen domains.

Findings

Achieves comparable performance to real data training on multiple datasets.

Demonstrates superior generalization on the RoboTool dataset with significant domain gap.

Uses hybrid augmentations to further diversify training data.

Abstract

Despite their impressive performance in various surgical scene understanding tasks, deep learning-based methods are frequently hindered from deploying to real-world surgical applications for various causes. Particularly, data collection, annotation, and domain shift in-between sites and patients are the most common obstacles. In this work, we mitigate data-related issues by efficiently leveraging minimal source images to generate synthetic surgical instrument segmentation datasets and achieve outstanding generalization performance on unseen real domains. Specifically, in our framework, only one background tissue image and at most three images of each foreground instrument are taken as the seed images. These source images are extensively transformed and employed to build up the foreground and background image pools, from which randomly sampled tissue and instrument images are composed with multiple blending techniques to generate new surgical scene images. Besides, we introduce hybrid training-time augmentations to diversify the training data further. Extensive evaluation on three real-world datasets, i.e., Endo2017, Endo2018, and RoboTool, demonstrates that our one-to-many synthetic surgical instruments datasets generation and segmentation framework can achieve encouraging performance compared with training with real data. Notably, on the RoboTool dataset, where a more significant domain gap exists, our framework shows its superiority of generalization by a considerable margin. We expect that our inspiring results will attract research attention to improving model generalization with data synthesizing.