SyncTwin: Fast Digital Twin Construction and Synchronization for Safe Robotic Manipulation

TL;DR

SyncTwin is a digital twin framework that enables fast scene reconstruction and synchronization, improving safety and robustness in robotic manipulation under occlusion and dynamic conditions.

Contribution

It introduces a novel real-time synchronization method combining rapid 3D reconstruction and continuous state updates for safe robotic manipulation.

Findings

Enhances manipulation performance in occluded scenes

Improves motion safety through synchronized digital twins

Demonstrates effectiveness in dynamic environments

Abstract

Accurate and safe robotic manipulation under dynamic and visually occluded conditions remains a core challenge in real-world deployment. We introduce SyncTwin, a novel digital twin framework that unifies fast 3D scene reconstruction and real-to-sim synchronization for robust and safety-aware robotic manipulation in such environments. In the offline stage, we employ VGGT to rapidly reconstruct object-level 3D assets from RGB images, forming a reusable geometry library. During execution, SyncTwin continuously synchronizes the digital twin by tracking real-world object states via point cloud segmentation updates and aligning them through colored-ICP registration. The synchronized twin enables motion planners to compute collision-free and dynamically feasible trajectories in simulation, which are safely executed on the real robot through a closed real-to-sim-to-real loop. Experiments in dynamic and occluded scenes show that SyncTwin improves manipulation performance and motion safety, demonstrating the effectiveness of digital twin synchronization for real-world robotic execution. The video demos and code can be found on the project website: https://sync-twin.github.io/.