Is Single-View Mesh Reconstruction Ready for Robotics?

TL;DR

This paper assesses whether current single-view 3D mesh reconstruction models are suitable for robotics applications, focusing on real-time digital twin creation and physics simulation readiness.

Contribution

It introduces robotics-specific benchmarking criteria for 3D reconstruction and evaluates existing models, revealing significant gaps for robotics use cases.

Findings

Existing models fail to meet robotics-specific requirements

Single-view reconstructions lack physical plausibility and stability

Current approaches are insufficient for real-time robotic simulation

Abstract

This paper evaluates single-view mesh reconstruction models for their potential in enabling instant digital twin creation for real-time planning and dynamics prediction using physics simulators for robotic manipulation. Recent single-view 3D reconstruction advances offer a promising avenue toward an automated real-to-sim pipeline: directly mapping a single observation of a scene into a simulation instance by reconstructing scene objects as individual, complete, and physically plausible 3D meshes. However, their suitability for physics simulations and robotics applications under immediacy, physical fidelity, and simulation readiness remains underexplored. We establish robotics-specific benchmarking criteria for 3D reconstruction, including handling typical inputs, collision-free and stable geometry, occlusions robustness, and meeting computational constraints. Our empirical evaluation using realistic robotics datasets shows that despite success on computer vision benchmarks, existing approaches fail to meet robotics-specific requirements. We quantitively examine limitations of single-view reconstruction for practical robotics implementation, in contrast to prior work that focuses on multi-view approaches. Our findings highlight critical gaps between computer vision advances and robotics needs, guiding future research at this intersection.