Part-level Scene Reconstruction Affords Robot Interaction

TL;DR

This paper presents a part-level scene reconstruction method that accurately replicates observed scenes using primitive shapes, enabling precise robot interaction simulation and improved scene understanding.

Contribution

The work introduces a novel part-level reconstruction approach that segments objects into semantic parts, aligns primitives, and estimates kinematic relations for more accurate scene modeling.

Findings

Outperforms object-level reconstruction in capturing scene details

Provides precise kinematic information for robotic applications

Enables validation of manipulation planning in reconstructed scenes

Abstract

Existing methods for reconstructing interactive scenes primarily focus on replacing reconstructed objects with CAD models retrieved from a limited database, resulting in significant discrepancies between the reconstructed and observed scenes. To address this issue, our work introduces a part-level reconstruction approach that reassembles objects using primitive shapes. This enables us to precisely replicate the observed physical scenes and simulate robot interactions with both rigid and articulated objects. By segmenting reconstructed objects into semantic parts and aligning primitive shapes to these parts, we assemble them as CAD models while estimating kinematic relations, including parent-child contact relations, joint types, and parameters. Specifically, we derive the optimal primitive alignment by solving a series of optimization problems, and estimate kinematic relations based on part semantics and geometry. Our experiments demonstrate that part-level scene reconstruction outperforms object-level reconstruction by accurately capturing finer details and improving precision. These reconstructed part-level interactive scenes provide valuable kinematic information for various robotic applications; we showcase the feasibility of certifying mobile manipulation planning in these interactive scenes before executing tasks in the physical world.