Instrument-Splatting++: Towards Controllable Surgical Instrument Digital Twin Using Gaussian Splatting

TL;DR

Instrument-Splatting++ introduces a controllable 3D Gaussian Splatting framework for surgical instruments, enabling high-fidelity reconstruction, pose estimation, and realistic texture learning from unposed endoscopic videos, advancing digital twin capabilities in robot-assisted surgery.

Contribution

The paper presents a novel Gaussian Splatting-based framework with part-aware geometry pretraining, semantics-aware pose estimation, and robust texture learning, enabling controllable and realistic surgical instrument digital twins from monocular videos.

Findings

Superior photometric quality over state-of-the-art methods

Improved geometric accuracy in instrument reconstruction

Enhanced keypoint detection performance with data augmentation

Abstract

High-quality and controllable digital twins of surgical instruments are critical for Real2Sim in robot-assisted surgery, as they enable realistic simulation, synthetic data generation, and perception learning under novel poses. We present Instrument-Splatting++, a monocular 3D Gaussian Splatting (3DGS) framework that reconstructs surgical instruments as a fully controllable Gaussian asset with high fidelity. Our pipeline starts with part-wise geometry pretraining that injects CAD priors into Gaussian primitives and equips the representation with part-aware semantic rendering. Built on the pretrained model, we propose a semantics-aware pose estimation and tracking (SAPET) method to recover per-frame 6-DoF pose and joint angles from unposed endoscopic videos, where a gripper-tip network trained purely from synthetic semantics provides robust supervision and a loose regularization suppresses singular articulations. Finally, we introduce Robust Texture Learning (RTL), which alternates pose refinement and robust appearance optimization, mitigating pose noise during texture learning. The proposed framework can perform pose estimation and learn realistic texture from unposed videos. We validate our method on sequences extracted from EndoVis17/18, SAR-RARP, and an in-house dataset, showing superior photometric quality and improved geometric accuracy over state-of-the-art baselines. We further demonstrate a downstream keypoint detection task where unseen-pose data augmentation from our controllable instrument Gaussian improves performance.