Efficient EndoNeRF Reconstruction and Its Application for Data-driven Surgical Simulation

TL;DR

This paper introduces a neural radiance field-based framework for fast, high-quality 3D reconstruction of surgical scenes, enabling realistic data-driven surgical simulations with improved accuracy and efficiency.

Contribution

It presents a novel neural radiance field approach for rapid, detailed surgical scene reconstruction and an automated pipeline for creating interactive simulation environments.

Findings

Outperforms traditional 3D reconstruction methods in capturing deformations and textures.

Achieves state-of-the-art performance in surgical scene 3D reconstruction.

Demonstrates practical application in FEM and MPM simulations for surgical training.

Abstract

The healthcare industry has a growing need for realistic modeling and efficient simulation of surgical scenes. With effective models of deformable surgical scenes, clinicians are able to conduct surgical planning and surgery training on scenarios close to real-world cases. However, a significant challenge in achieving such a goal is the scarcity of high-quality soft tissue models with accurate shapes and textures. To address this gap, we present a data-driven framework that leverages emerging neural radiance field technology to enable high-quality surgical reconstruction and explore its application for surgical simulations. We first focus on developing a fast NeRF-based surgical scene 3D reconstruction approach that achieves state-of-the-art performance. This method can significantly outperform traditional 3D reconstruction methods, which have failed to capture large deformations and produce fine-grained shapes and textures. We then propose an automated creation pipeline of interactive surgical simulation environments through a closed mesh extraction algorithm. Our experiments have validated the superior performance and efficiency of our proposed approach in surgical scene 3D reconstruction. We further utilize our reconstructed soft tissues to conduct FEM and MPM simulations, showcasing the practical application of our method in data-driven surgical simulations.