SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment

TL;DR

SIU3R introduces an alignment-free, unified framework for simultaneous scene understanding and 3D reconstruction from unposed images, surpassing previous methods that relied on feature alignment.

Contribution

It is the first to eliminate feature alignment in joint 3D reconstruction and understanding, enabling native 3D understanding and improved task collaboration.

Findings

Achieves state-of-the-art results on 3D reconstruction and understanding tasks.

Demonstrates superior performance on simultaneous scene understanding and 3D reconstruction.

Validates the effectiveness of mutual benefit modules in task collaboration.

Abstract

Simultaneous understanding and 3D reconstruction plays an important role in developing end-to-end embodied intelligent systems. To achieve this, recent approaches resort to 2D-to-3D feature alignment paradigm, which leads to limited 3D understanding capability and potential semantic information loss. In light of this, we propose SIU3R, the first alignment-free framework for generalizable simultaneous understanding and 3D reconstruction from unposed images. Specifically, SIU3R bridges reconstruction and understanding tasks via pixel-aligned 3D representation, and unifies multiple understanding (segmentation) tasks into a set of unified learnable queries, enabling native 3D understanding without the need of alignment with 2D models. To encourage collaboration between the two tasks with shared representation, we further conduct in-depth analyses of their mutual benefits, and propose two lightweight modules to facilitate their interaction. Extensive experiments demonstrate that our method achieves state-of-the-art performance not only on the individual tasks of 3D reconstruction and understanding, but also on the task of simultaneous understanding and 3D reconstruction, highlighting the advantages of our alignment-free framework and the effectiveness of the mutual benefit designs. Project page: https://insomniaaac.github.io/siu3r/