GaussTwin: Unified Simulation and Correction with Gaussian Splatting for Robotic Digital Twins

TL;DR

GaussTwin introduces a real-time digital twin framework that combines physics-based simulation with Gaussian splatting for accurate, robust robotic tracking and manipulation, bridging the gap between perception and simulation.

Contribution

It presents GaussTwin, a novel unified system integrating physics simulation and visual correction for robotic digital twins, improving accuracy and robustness.

Findings

Enhanced tracking accuracy over baselines

Robustness in dynamic and complex interactions

Supports downstream robotic tasks like planning

Abstract

Digital twins promise to enhance robotic manipulation by maintaining a consistent link between real-world perception and simulation. However, most existing systems struggle with the lack of a unified model, complex dynamic interactions, and the real-to-sim gap, which limits downstream applications such as model predictive control. Thus, we propose GaussTwin, a real-time digital twin that combines position-based dynamics with discrete Cosserat rod formulations for physically grounded simulation, and Gaussian splatting for efficient rendering and visual correction. By anchoring Gaussians to physical primitives and enforcing coherent SE(3) updates driven by photometric error and segmentation masks, GaussTwin achieves stable prediction-correction while preserving physical fidelity. Through experiments in both simulation and on a Franka Research 3 platform, we show that GaussTwin consistently improves tracking accuracy and robustness compared to shape-matching and rigid-only baselines, while also enabling downstream tasks such as push-based planning. These results highlight GaussTwin as a step toward unified, physically meaningful digital twins that can support closed-loop robotic interaction and learning.