Tech Report: Divide and Conquer 3D Real-Time Reconstruction for Improved IGS

TL;DR

This paper introduces a modular pipeline for real-time 3D reconstruction in surgical settings, integrating advanced depth estimation and alignment methods to enhance accuracy and flexibility based on endoscopic videos.

Contribution

The paper presents a flexible, modular pipeline combining recent depth estimation and alignment techniques for improved real-time 3D reconstruction in surgical applications.

Findings

Effective integration of Depth-Anything V2 and EndoDAC for depth estimation

Enhanced ICP alignment process improves reconstruction accuracy

Demonstrated system performance on Hamlyn dataset

Abstract

Tracking surgical modifications based on endoscopic videos is technically feasible and of great clinical advantages; however, it still remains challenging. This report presents a modular pipeline to divide and conquer the clinical challenges in the process. The pipeline integrates frame selection, depth estimation, and 3D reconstruction components, allowing for flexibility and adaptability in incorporating new methods. Recent advancements, including the integration of Depth-Anything V2 and EndoDAC for depth estimation, as well as improvements in the Iterative Closest Point (ICP) alignment process, are detailed. Experiments conducted on the Hamlyn dataset demonstrate the effectiveness of the integrated methods. System capability and limitations are both discussed.