PlaneRCNN: 3D Plane Detection and Reconstruction from a Single Image

TL;DR

PlaneRCNN is a deep neural network that detects and reconstructs 3D planes from a single RGB image, improving accuracy and robustness over previous methods, with applications in AR, VR, and robotics.

Contribution

The paper introduces PlaneRCNN, a novel architecture with a new loss and benchmark for improved 3D plane detection and reconstruction from single images.

Findings

Outperforms existing methods in plane detection and segmentation

Achieves significant improvements in reconstruction accuracy

Provides a new benchmark with fine-grained plane annotations

Abstract

This paper proposes a deep neural architecture, PlaneRCNN, that detects and reconstructs piecewise planar surfaces from a single RGB image. PlaneRCNN employs a variant of Mask R-CNN to detect planes with their plane parameters and segmentation masks. PlaneRCNN then jointly refines all the segmentation masks with a novel loss enforcing the consistency with a nearby view during training. The paper also presents a new benchmark with more fine-grained plane segmentations in the ground-truth, in which, PlaneRCNN outperforms existing state-of-the-art methods with significant margins in the plane detection, segmentation, and reconstruction metrics. PlaneRCNN makes an important step towards robust plane extraction, which would have an immediate impact on a wide range of applications including Robotics, Augmented Reality, and Virtual Reality.