Splatting Physical Scenes: End-to-End Real-to-Sim from Imperfect Robot Data

TL;DR

This paper presents a novel end-to-end framework that converts imperfect real robot data into accurate, photorealistic digital twins suitable for physics simulation, combining differentiable rendering and physics for joint refinement.

Contribution

Introduces a hybrid scene representation and an end-to-end optimization pipeline that jointly refines geometry, appearance, and robot poses from noisy real-world data.

Findings

High-fidelity object mesh reconstruction achieved

Photorealistic novel view synthesis demonstrated

Robot pose calibration performed without annotations

Abstract

Creating accurate, physical simulations directly from real-world robot motion holds great value for safe, scalable, and affordable robot learning, yet remains exceptionally challenging. Real robot data suffers from occlusions, noisy camera poses, dynamic scene elements, which hinder the creation of geometrically accurate and photorealistic digital twins of unseen objects. We introduce a novel real-to-sim framework tackling all these challenges at once. Our key insight is a hybrid scene representation merging the photorealistic rendering of 3D Gaussian Splatting with explicit object meshes suitable for physics simulation within a single representation. We propose an end-to-end optimization pipeline that leverages differentiable rendering and differentiable physics within MuJoCo to jointly refine all scene components - from object geometry and appearance to robot poses and physical parameters - directly from raw and imprecise robot trajectories. This unified optimization allows us to simultaneously achieve high-fidelity object mesh reconstruction, generate photorealistic novel views, and perform annotation-free robot pose calibration. We demonstrate the effectiveness of our approach both in simulation and on challenging real-world sequences using an ALOHA 2 bi-manual manipulator, enabling more practical and robust real-to-simulation pipelines.