Learning Spatial Awareness for Laparoscopic Surgery with AI Assisted Visual Feedback

TL;DR

This paper introduces an AI-assisted visual feedback system for laparoscopic surgery training that enhances spatial awareness by providing synchronized 3D visual cues alongside standard 2D views, improving depth perception and instrument handling.

Contribution

The study presents a novel mixed-reality framework integrating AI modules for real-time instrument localization and interaction detection with 3D feedback, enhancing surgical training effectiveness.

Findings

Improved depth perception and spatial understanding in simulated surgical tasks.

Enhanced accuracy in instrument positioning and tissue interaction detection.

Effective disambiguation of similar 2D cases with 3D visual cues.

Abstract

Laparoscopic surgery constrains surgeons spatial awareness because procedures are performed through a monocular, two-dimensional (2D) endoscopic view. Conventional training methods using dry-lab models or recorded videos provide limited depth cues, often leading trainees to misjudge instrument position and perform ineffective or unsafe maneuvers. To address this limitation, we present an AI-assisted training framework developed in NVIDIA Isaac Sim that couples the standard 2D laparoscopic feed with synchronized three-dimensional (3D) visual feedback delivered through a mixed-reality (MR) interface. While trainees operate using the clinical 2D view, validated AI modules continuously localize surgical instruments and detect instrument-tissue interactions in the background. When spatial misjudgments are detected, 3D visual feedback are displayed to trainees, while preserving the original operative perspective. Our framework considers various surgical tasks including navigation, manipulation, transfer, cutting, and suturing. Visually similar 2D cases can be disambiguated through the added 3D context, improving depth perception, contact awareness, and tool orientation understanding.