SurgeoNet: Realtime 3D Pose Estimation of Articulated Surgical Instruments from Stereo Images using a Synthetically-trained Network

TL;DR

SurgeoNet is a real-time neural network system that accurately detects and tracks articulated surgical instruments from stereo images, trained solely on synthetic data, enhancing surgical monitoring in mixed reality environments.

Contribution

This work introduces SurgeoNet, a novel synthetic-data-trained neural network pipeline for real-time 3D pose estimation of surgical instruments from stereo images.

Findings

Effective generalization to real-world scenarios

Real-time performance suitable for surgical applications

Flexible extension to various surgical instruments

Abstract

Surgery monitoring in Mixed Reality (MR) environments has recently received substantial focus due to its importance in image-based decisions, skill assessment, and robot-assisted surgery. Tracking hands and articulated surgical instruments is crucial for the success of these applications. Due to the lack of annotated datasets and the complexity of the task, only a few works have addressed this problem. In this work, we present SurgeoNet, a real-time neural network pipeline to accurately detect and track surgical instruments from a stereo VR view. Our multi-stage approach is inspired by state-of-the-art neural-network architectural design, like YOLO and Transformers. We demonstrate the generalization capabilities of SurgeoNet in challenging real-world scenarios, achieved solely through training on synthetic data. The approach can be easily extended to any new set of articulated surgical instruments. SurgeoNet's code and data are publicly available.