PlaneRecTR++: Unified Query Learning for Joint 3D Planar Reconstruction and Pose Estimation

TL;DR

PlaneRecTR++ introduces a unified Transformer-based framework that simultaneously performs 3D planar reconstruction and pose estimation from images, eliminating the need for separate modules and initial pose estimation, leading to state-of-the-art results.

Contribution

It is the first to unify all sub-tasks of multi-view planar reconstruction and pose estimation into a single-stage, query-based Transformer framework without external supervision.

Findings

Achieves new state-of-the-art performance on multiple datasets.

Eliminates the need for initial pose estimation and external plane correspondence labels.

Demonstrates mutual benefits across sub-tasks through unified learning.

Abstract

The challenging task of 3D planar reconstruction from images involves several sub-tasks including frame-wise plane detection, segmentation, parameter regression and possibly depth prediction, along with cross-frame plane correspondence and relative camera pose estimation. Previous works adopt a divide and conquer strategy, addressing above sub-tasks with distinct network modules in a two-stage paradigm. Specifically, given an initial camera pose and per-frame plane predictions from the first stage, further exclusively designed modules relying on external plane correspondence labeling are applied to merge multi-view plane entities and produce refined camera pose. Notably, existing work fails to integrate these closely related sub-tasks into a unified framework, and instead addresses them separately and sequentially, which we identify as a primary source of performance limitations. Motivated by this finding and the success of query-based learning in enriching reasoning among semantic entities, in this paper, we propose PlaneRecTR++, a Transformer-based architecture, which for the first time unifies all tasks of multi-view planar reconstruction and pose estimation within a compact single-stage framework, eliminating the need for the initial pose estimation and supervision of plane correspondence. Extensive quantitative and qualitative experiments demonstrate that our proposed unified learning achieves mutual benefits across sub-tasks, achieving a new state-of-the-art performance on the public ScanNetv1, ScanNetv2, NYUv2-Plane, and MatterPort3D datasets. Codes are available at https://github.com/SJingjia/PlaneRecTR-PP.