Synthetic and Real Inputs for Tool Segmentation in Robotic Surgery

TL;DR

This paper introduces a novel deep learning model that combines laparoscopic images and robot kinematic data to improve surgical tool segmentation robustness, addressing data scarcity with a new custom dataset.

Contribution

It presents a new model for simultaneous processing of laparoscopic and simulation images, and provides a custom dataset for training and evaluation.

Findings

Enhanced segmentation robustness under challenging conditions

Successful integration of kinematic data with imaging

Availability of a new annotated dataset for research

Abstract

Semantic tool segmentation in surgical videos is important for surgical scene understanding and computer-assisted interventions as well as for the development of robotic automation. The problem is challenging because different illumination conditions, bleeding, smoke and occlusions can reduce algorithm robustness. At present labelled data for training deep learning models is still lacking for semantic surgical instrument segmentation and in this paper we show that it may be possible to use robot kinematic data coupled with laparoscopic images to alleviate the labelling problem. We propose a new deep learning based model for parallel processing of both laparoscopic and simulation images for robust segmentation of surgical tools. Due to the lack of laparoscopic frames annotated with both segmentation ground truth and kinematic information a new custom dataset was generated using the da Vinci Research Kit (dVRK) and is made available.