Neural Rendering for Stereo 3D Reconstruction of Deformable Tissues in Robotic Surgery

TL;DR

This paper introduces a neural rendering framework using dynamic neural radiance fields for reconstructing deformable tissues in robotic surgery from stereo videos, addressing challenges like non-rigid deformations and occlusions.

Contribution

It presents the first neural rendering approach for surgical scene 3D reconstruction, improving accuracy over existing methods in complex deformable tissue scenarios.

Findings

Outperforms state-of-the-art reconstruction methods on DaVinci surgical videos.

Effectively handles non-rigid tissue deformations and tool occlusions.

Demonstrates potential for intra-operative navigation and robotic surgery automation.

Abstract

Reconstruction of the soft tissues in robotic surgery from endoscopic stereo videos is important for many applications such as intra-operative navigation and image-guided robotic surgery automation. Previous works on this task mainly rely on SLAM-based approaches, which struggle to handle complex surgical scenes. Inspired by recent progress in neural rendering, we present a novel framework for deformable tissue reconstruction from binocular captures in robotic surgery under the single-viewpoint setting. Our framework adopts dynamic neural radiance fields to represent deformable surgical scenes in MLPs and optimize shapes and deformations in a learning-based manner. In addition to non-rigid deformations, tool occlusion and poor 3D clues from a single viewpoint are also particular challenges in soft tissue reconstruction. To overcome these difficulties, we present a series of strategies of tool mask-guided ray casting, stereo depth-cueing ray marching and stereo depth-supervised optimization. With experiments on DaVinci robotic surgery videos, our method significantly outperforms the current state-of-the-art reconstruction method for handling various complex non-rigid deformations. To our best knowledge, this is the first work leveraging neural rendering for surgical scene 3D reconstruction with remarkable potential demonstrated. Code is available at: https://github.com/med-air/EndoNeRF.