Using Conditional Generative Adversarial Networks to Reduce the Effects of Latency in Robotic Telesurgery

TL;DR

This paper presents a deep learning approach using conditional GANs to accurately detect robotic surgical tools in real-time, aiming to mitigate latency effects and enhance safety in telesurgery.

Contribution

The study introduces a purely optical, neural network-based method for real-time tool detection in robotic surgery, improving safety by reducing latency-related risks.

Findings

Achieved near-perfect label generation in 299 ms

Demonstrated the potential for real-time tool monitoring

Validated the approach on surgical data from the EndoVis challenge

Abstract

The introduction of surgical robots brought about advancements in surgical procedures. The applications of remote telesurgery range from building medical clinics in underprivileged areas, to placing robots abroad in military hot-spots where accessibility and diversity of medical experience may be limited. Poor wireless connectivity may result in a prolonged delay, referred to as latency, between a surgeon's input and action a robot takes. In surgery, any micro-delay can injure a patient severely and in some cases, result in fatality. One was to increase safety is to mitigate the effects of latency using deep learning aided computer vision. While the current surgical robots use calibrated sensors to measure the position of the arms and tools, in this work we present a purely optical approach that provides a measurement of the tool position in relation to the patient's tissues. This research aimed to produce a neural network that allowed a robot to detect its own mechanical manipulator arms. A conditional generative adversarial networks (cGAN) was trained on 1107 frames of mock gastrointestinal robotic surgery data from the 2015 EndoVis Instrument Challenge and corresponding hand-drawn labels for each frame. When run on new testing data, the network generated near-perfect labels of the input images which were visually consistent with the hand-drawn labels and was able to do this in 299 milliseconds. These accurately generated labels can then be used as simplified identifiers for the robot to track its own controlled tools. These results show potential for conditional GANs as a reaction mechanism such that the robot can detect when its arms move outside the operating area within a patient. This system allows for more accurate monitoring of the position of surgical instruments in relation to the patient's tissue, increasing safety measures that are integral to successful telesurgery systems.