Integrating Artificial Intelligence and Augmented Reality in Robotic Surgery: An Initial dVRK Study Using a Surgical Education Scenario

TL;DR

This paper presents a novel system integrating AI and AR in robotic surgery education, using reinforcement learning to generate 3D guidance trajectories overlaid in stereo views, enhancing surgical training.

Contribution

It introduces a new system that combines reinforcement learning, AI, and AR visualization for robotic surgery education, which has not been previously explored.

Findings

Feasibility demonstrated in peg-transfer task

Enhanced visualization aids surgical learning

Potential for next-generation surgical education tools

Abstract

Robot-assisted surgery has become progressively more and more popular due to its clinical advantages. In the meanwhile, the artificial intelligence and augmented reality in robotic surgery are developing rapidly and receive lots of attention. However, current methods have not discussed the coherent integration of AI and AR in robotic surgery. In this paper, we develop a novel system by seamlessly merging artificial intelligence module and augmented reality visualization to automatically generate the surgical guidance for robotic surgery education. Specifically, we first leverage reinforcement leaning to learn from expert demonstration and then generate 3D guidance trajectory, providing prior context information of the surgical procedure. Along with other information such as text hint, the 3D trajectory is then overlaid in the stereo view of dVRK, where the user can perceive the 3D guidance and learn the procedure. The proposed system is evaluated through a preliminary experiment on surgical education task peg-transfer, which proves its feasibility and potential as the next generation of robot-assisted surgery education solution.