Scan, Materialize, Simulate: A Generalizable Framework for Physically Grounded Robot Planning

TL;DR

This paper introduces SMS, a unified framework combining scene reconstruction, semantic understanding, material inference, and physics simulation to enable robots to reason about physical consequences in unstructured environments.

Contribution

The paper presents SMS, a novel integrated approach that leverages 3D Gaussian Splatting, foundation models, and physics simulation for generalizable robot planning without re-learning physical dynamics.

Findings

Demonstrates robust performance in manipulation and quadrotor landing tasks.

Effective domain transfer from simulation to real-world scenarios.

Bridges differentiable rendering, foundation models, and physics simulation for physical reasoning.

Abstract

Autonomous robots must reason about the physical consequences of their actions to operate effectively in unstructured, real-world environments. We present Scan, Materialize, Simulate (SMS), a unified framework that combines 3D Gaussian Splatting for accurate scene reconstruction, visual foundation models for semantic segmentation, vision-language models for material property inference, and physics simulation for reliable prediction of action outcomes. By integrating these components, SMS enables generalizable physical reasoning and object-centric planning without the need to re-learn foundational physical dynamics. We empirically validate SMS in a billiards-inspired manipulation task and a challenging quadrotor landing scenario, demonstrating robust performance on both simulated domain transfer and real-world experiments. Our results highlight the potential of bridging differentiable rendering for scene reconstruction, foundation models for semantic understanding, and physics-based simulation to achieve physically grounded robot planning across diverse settings.