PointRecon: Online Point-based 3D Reconstruction via Ray-based 2D-3D Matching

TL;DR

PointRecon introduces a real-time, online point-based 3D reconstruction method from monocular videos that maintains a global scene point cloud using a novel ray-based 2D-3D matching technique, enabling continuous updates and view consistency.

Contribution

It presents a novel online point cloud reconstruction approach with ray-based matching that handles infinite sequences and does not require scene size constraints.

Findings

Achieves comparable quality to offline methods on ScanNet

Supports real-time, continuous scene updates

Maintains view-consistent global point cloud

Abstract

We propose a novel online, point-based 3D reconstruction method from posed monocular RGB videos. Our model maintains a global point cloud representation of the scene, continuously updating the features and 3D locations of points as new images are observed. It expands the point cloud with newly detected points while carefully removing redundancies. The point cloud updates and the depth predictions for new points are achieved through a novel ray-based 2D-3D feature matching technique, which is robust against errors in previous point position predictions. In contrast to offline methods, our approach processes infinite-length sequences and provides real-time updates. Additionally, the point cloud imposes no pre-defined resolution or scene size constraints, and its unified global representation ensures view consistency across perspectives. Experiments on the ScanNet dataset show that our method achieves comparable quality among online MVS approaches. Project page: https://arthurhero.github.io/projects/pointrecon