PlaneTR: Structure-Guided Transformers for 3D Plane Recovery

TL;DR

PlaneTR is a Transformer-based neural network that detects and reconstructs 3D planes from a single image by jointly leveraging contextual and structural information in a sequence-to-sequence framework, achieving state-of-the-art results.

Contribution

The paper introduces PlaneTR, a novel Transformer architecture that integrates geometric structures and context for holistic 3D plane detection and reconstruction from images.

Findings

Achieves state-of-the-art performance on ScanNet and NYUv2 datasets.

Effectively combines structural cues and context in a unified Transformer framework.

Outperforms previous methods in accuracy and efficiency.

Abstract

This paper presents a neural network built upon Transformers, namely PlaneTR, to simultaneously detect and reconstruct planes from a single image. Different from previous methods, PlaneTR jointly leverages the context information and the geometric structures in a sequence-to-sequence way to holistically detect plane instances in one forward pass. Specifically, we represent the geometric structures as line segments and conduct the network with three main components: (i) context and line segments encoders, (ii) a structure-guided plane decoder, (iii) a pixel-wise plane embedding decoder. Given an image and its detected line segments, PlaneTR generates the context and line segment sequences via two specially designed encoders and then feeds them into a Transformers-based decoder to directly predict a sequence of plane instances by simultaneously considering the context and global structure cues. Finally, the pixel-wise embeddings are computed to assign each pixel to one predicted plane instance which is nearest to it in embedding space. Comprehensive experiments demonstrate that PlaneTR achieves a state-of-the-art performance on the ScanNet and NYUv2 datasets.