SAMSNeRF: Segment Anything Model (SAM) Guides Dynamic Surgical Scene Reconstruction by Neural Radiance Field (NeRF)

TL;DR

SAMSNeRF integrates Segment Anything Model with Neural Radiance Field to improve 3D reconstruction of dynamic surgical scenes, accurately capturing surgical tools' positions for enhanced navigation and automation.

Contribution

This paper introduces a novel method combining SAM and NeRF to accurately reconstruct surgical scenes with moving tools, surpassing previous depth-based approaches.

Findings

Successfully reconstructs high-fidelity dynamic surgical scenes

Accurately predicts 3D positions of surgical tools

Enhances surgical navigation and automation

Abstract

The accurate reconstruction of surgical scenes from surgical videos is critical for various applications, including intraoperative navigation and image-guided robotic surgery automation. However, previous approaches, mainly relying on depth estimation, have limited effectiveness in reconstructing surgical scenes with moving surgical tools. To address this limitation and provide accurate 3D position prediction for surgical tools in all frames, we propose a novel approach called SAMSNeRF that combines Segment Anything Model (SAM) and Neural Radiance Field (NeRF) techniques. Our approach generates accurate segmentation masks of surgical tools using SAM, which guides the refinement of the dynamic surgical scene reconstruction by NeRF. Our experimental results on public endoscopy surgical videos demonstrate that our approach successfully reconstructs high-fidelity dynamic surgical scenes and accurately reflects the spatial information of surgical tools. Our proposed approach can significantly enhance surgical navigation and automation by providing surgeons with accurate 3D position information of surgical tools during surgery.The source code will be released soon.