Localising under the drape: proprioception in the era of distributed surgical robotic system

TL;DR

This paper introduces a marker-free, deep learning-based proprioception method for surgical robots under drapes, enabling precise localization without bulky hardware, thus improving safety and workflow in robotic surgery.

Contribution

It presents the first marker-free proprioception approach using stereo-RGB cameras and transformer models, leveraging the largest multi-centre dataset for surgical robot localization.

Findings

Achieved 25% improvement in tracking visibility over existing systems.

Demonstrated clinical benefits in breathing compensation and multi-robot localization.

Reduced setup complexity and enhanced safety in robotic surgery.

Abstract

Despite their mechanical sophistication, surgical robots remain blind to their surroundings. This lack of spatial awareness causes collisions, system recoveries, and workflow disruptions, issues that will intensify with the introduction of distributed robots with independent interacting arms. Existing tracking systems rely on bulky infrared cameras and reflective markers, providing only limited views of the surgical scene and adding hardware burden in crowded operating rooms. We present a marker-free proprioception method that enables precise localisation of surgical robots under their sterile draping despite associated obstruction of visual cues. Our method solely relies on lightweight stereo-RGB cameras and novel transformer-based deep learning models. It builds on the largest multi-centre spatial robotic surgery dataset to date (1.4M self-annotated images from human cadaveric and preclinical in vivo studies). By tracking the entire robot and surgical scene, rather than individual markers, our approach provides a holistic view robust to occlusions, supporting surgical scene understanding and context-aware control. We demonstrate an example of potential clinical benefits during in vivo breathing compensation with access to tissue dynamics, unobservable under state of the art tracking, and accurately locate in multi-robot systems for future intelligent interaction. In addition, and compared with existing systems, our method eliminates markers and improves tracking visibility by 25%. To our knowledge, this is the first demonstration of marker-free proprioception for fully draped surgical robots, reducing setup complexity, enhancing safety, and paving the way toward modular and autonomous robotic surgery.