Real-to-Sim for Highly Cluttered Environments via Physics-Consistent Inter-Object Reasoning

TL;DR

This paper introduces a physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data, improving robotic manipulation in cluttered environments.

Contribution

It presents a novel differentiable optimization approach that models spatial dependencies with a contact graph to refine object poses and physical properties jointly.

Findings

Reconstructed scenes exhibit high physical fidelity in simulation and real-world tests.

The approach enables stable, contact-rich manipulation in cluttered environments.

Extensive evaluations validate the effectiveness of the method in diverse scenarios.

Abstract

Reconstructing physically valid 3D scenes from single-view observations is a prerequisite for bridging the gap between visual perception and robotic control. However, in scenarios requiring precise contact reasoning, such as robotic manipulation in highly cluttered environments, geometric fidelity alone is insufficient. Standard perception pipelines often neglect physical constraints, resulting in invalid states, e.g., floating objects or severe inter-penetration, rendering downstream simulation unreliable. To address these limitations, we propose a novel physics-constrained Real-to-Sim pipeline that reconstructs physically consistent 3D scenes from single-view RGB-D data. Central to our approach is a differentiable optimization pipeline that explicitly models spatial dependencies via a contact graph, jointly refining object poses and physical properties through differentiable rigid-body simulation. Extensive evaluations in both simulation and real-world settings demonstrate that our reconstructed scenes achieve high physical fidelity and faithfully replicate real-world contact dynamics, enabling stable and reliable contact-rich manipulation.