Free-SurGS: SfM-Free 3D Gaussian Splatting for Surgical Scene Reconstruction

TL;DR

This paper introduces Free-SurGS, a novel SfM-free 3D Gaussian Splatting method for surgical scene reconstruction that jointly optimizes camera poses and scene representation using optical flow priors, improving accuracy and efficiency.

Contribution

It presents the first SfM-free approach for surgical scene reconstruction using 3D Gaussian Splatting, leveraging optical flow to guide pose estimation without relying on structure-from-motion.

Findings

Outperforms existing methods in novel view synthesis

Achieves higher accuracy in pose estimation

Demonstrates high efficiency on surgical datasets

Abstract

Real-time 3D reconstruction of surgical scenes plays a vital role in computer-assisted surgery, holding a promise to enhance surgeons' visibility. Recent advancements in 3D Gaussian Splatting (3DGS) have shown great potential for real-time novel view synthesis of general scenes, which relies on accurate poses and point clouds generated by Structure-from-Motion (SfM) for initialization. However, 3DGS with SfM fails to recover accurate camera poses and geometry in surgical scenes due to the challenges of minimal textures and photometric inconsistencies. To tackle this problem, in this paper, we propose the first SfM-free 3DGS-based method for surgical scene reconstruction by jointly optimizing the camera poses and scene representation. Based on the video continuity, the key of our method is to exploit the immediate optical flow priors to guide the projection flow derived from 3D Gaussians. Unlike most previous methods relying on photometric loss only, we formulate the pose estimation problem as minimizing the flow loss between the projection flow and optical flow. A consistency check is further introduced to filter the flow outliers by detecting the rigid and reliable points that satisfy the epipolar geometry. During 3D Gaussian optimization, we randomly sample frames to optimize the scene representations to grow the 3D Gaussian progressively. Experiments on the SCARED dataset demonstrate our superior performance over existing methods in novel view synthesis and pose estimation with high efficiency. Code is available at https://github.com/wrld/Free-SurGS.