PlanarRecon: Real-time 3D Plane Detection and Reconstruction from Posed Monocular Videos

TL;DR

PlanarRecon is a real-time framework that detects and reconstructs 3D planes from monocular videos by incrementally integrating multi-view observations into a coherent scene model.

Contribution

It introduces a novel incremental 3D plane detection method from monocular videos using neural networks and a fusion module for global scene coherence.

Findings

Achieves state-of-the-art results on ScanNet dataset

Operates in real-time with low-polygonal geometry

Effectively integrates multi-view observations for accurate scene reconstruction

Abstract

We present PlanarRecon -- a novel framework for globally coherent detection and reconstruction of 3D planes from a posed monocular video. Unlike previous works that detect planes in 2D from a single image, PlanarRecon incrementally detects planes in 3D for each video fragment, which consists of a set of key frames, from a volumetric representation of the scene using neural networks. A learning-based tracking and fusion module is designed to merge planes from previous fragments to form a coherent global plane reconstruction. Such design allows PlanarRecon to integrate observations from multiple views within each fragment and temporal information across different ones, resulting in an accurate and coherent reconstruction of the scene abstraction with low-polygonal geometry. Experiments show that the proposed approach achieves state-of-the-art performances on the ScanNet dataset while being real-time.