Real-time Monocular 2D and 3D Perception of Endoluminal Scenes for Controlling Flexible Robotic Endoscopic Instruments

TL;DR

This paper develops a perception system using monocular endoscopic images and simulation to improve control of flexible robotic instruments in minimally invasive endoluminal surgeries, achieving significant reductions in manipulation time.

Contribution

It introduces novel 2D and 3D perception algorithms and a realistic simulator for flexible endoluminal robots, enhancing surgical control and data collection capabilities.

Findings

Reduced manipulation time by over 70% in trajectory tasks

Improved control accuracy of flexible instruments

Enhanced understanding of surgical scenarios

Abstract

Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract cancers but is limited by surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise tissue resection, potentially improving outcomes. This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery. Our goal is to utilize monocular endoscopic image-based perception algorithms to identify position and orientation of flexible instruments and measure their distances from tissues. We introduce 2D and 3D learning-based perception algorithms and develop a physically-realistic simulator that models flexible instruments dynamics. This simulator generates realistic endoluminal scenes, enabling control of flexible robots and substantial data collection. Using a continuum robot prototype, we conducted module and system-level evaluations. Results show that our algorithms improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.