EndoGSim: Physics-Aware 4D Dynamic Endoscopic Scene Simulations via MLLM-Guided Gaussian Splatting

TL;DR

This paper introduces EndoGSim, a physics-aware 4D endoscopic scene simulation framework that combines Gaussian Splatting with MLLM-guided material inference for realistic surgical scene modeling.

Contribution

It presents a novel unified approach integrating physics-based scene reconstruction and simulation using MLLMs and Gaussian Splatting, advancing surgical scene modeling.

Findings

Achieves higher simulation fidelity than existing methods.

Accurately models deformable tissues and tools.

Demonstrates effectiveness on multiple datasets.

Abstract

In robot-assisted minimally invasive surgery, high-fidelity dynamic endoscopic scene reconstruction and simulation are crucial to enhancing downstream tasks and advancing surgical outcomes. However, existing methods primarily focus on visual reconstruction, lacking physics-based descriptions of the scene required for realistic simulation. We propose a unified framework that achieves physics-aware reconstruction and physical simulation of endoscopic scenes through Multi-modal Large Language Models (MLLMs)-guided Gaussian Splatting. Our approach utilizes 4D Gaussian Splatting (4DGS) integrated with pre-trained segmentation and depth estimation to represent deformable tissues and tools. To achieve automatic inference of physical properties, we introduce an object-wise material field that initializes material parameters via MLLM and refines them through a differentiable Material Point Method (MPM) under joint supervision from rendered images and optical flow. Validated on both open-source and in-house datasets, our framework achieves superior simulation fidelity and physical accuracy compared to state-of-the-art methods, underscoring its potential to advance robot-assisted surgical applications.