NOPE-SAC: Neural One-Plane RANSAC for Sparse-View Planar 3D Reconstruction

TL;DR

This paper introduces NOPE-SAC, a neural RANSAC framework that improves two-view sparse-view 3D reconstruction by learning one-plane pose hypotheses from 3D plane correspondences, achieving state-of-the-art results.

Contribution

The paper proposes a novel neural one-plane RANSAC method that enhances pose estimation in sparse-view scenarios by leveraging learned plane correspondences and adaptive hypotheses.

Findings

Significantly improves camera pose estimation accuracy.

Achieves state-of-the-art performance on MatterPort3D and ScanNet benchmarks.

Enables stable pose estimation with minimal plane correspondences.

Abstract

This paper studies the challenging two-view 3D reconstruction in a rigorous sparse-view configuration, which is suffering from insufficient correspondences in the input image pairs for camera pose estimation. We present a novel Neural One-PlanE RANSAC framework (termed NOPE-SAC in short) that exerts excellent capability to learn one-plane pose hypotheses from 3D plane correspondences. Building on the top of a siamese plane detection network, our NOPE-SAC first generates putative plane correspondences with a coarse initial pose. It then feeds the learned 3D plane parameters of correspondences into shared MLPs to estimate the one-plane camera pose hypotheses, which are subsequently reweighed in a RANSAC manner to obtain the final camera pose. Because the neural one-plane pose minimizes the number of plane correspondences for adaptive pose hypotheses generation, it enables stable pose voting and reliable pose refinement in a few plane correspondences for the sparse-view inputs. In the experiments, we demonstrate that our NOPE-SAC significantly improves the camera pose estimation for the two-view inputs with severe viewpoint changes, setting several new state-of-the-art performances on two challenging benchmarks, i.e., MatterPort3D and ScanNet, for sparse-view 3D reconstruction. The source code is released at https://github.com/IceTTTb/NopeSAC for reproducible research.