SimpleRecon: 3D Reconstruction Without 3D Convolutions

TL;DR

SimpleRecon demonstrates that high-quality multi-view depth prediction combined with simple depth fusion can achieve accurate 3D scene reconstructions without expensive 3D convolutions, enabling real-time applications.

Contribution

The paper introduces a novel multi-view depth estimator that leverages image priors, geometric losses, and metadata integration, surpassing current methods in accuracy while maintaining efficiency.

Findings

Achieves state-of-the-art depth estimation results.

Provides accurate 3D reconstructions comparable to or better than existing methods.

Enables real-time, low-memory 3D reconstruction.

Abstract

Traditionally, 3D indoor scene reconstruction from posed images happens in two phases: per-image depth estimation, followed by depth merging and surface reconstruction. Recently, a family of methods have emerged that perform reconstruction directly in final 3D volumetric feature space. While these methods have shown impressive reconstruction results, they rely on expensive 3D convolutional layers, limiting their application in resource-constrained environments. In this work, we instead go back to the traditional route, and show how focusing on high quality multi-view depth prediction leads to highly accurate 3D reconstructions using simple off-the-shelf depth fusion. We propose a simple state-of-the-art multi-view depth estimator with two main contributions: 1) a carefully-designed 2D CNN which utilizes strong image priors alongside a plane-sweep feature volume and geometric losses, combined with 2) the integration of keyframe and geometric metadata into the cost volume which allows informed depth plane scoring. Our method achieves a significant lead over the current state-of-the-art for depth estimation and close or better for 3D reconstruction on ScanNet and 7-Scenes, yet still allows for online real-time low-memory reconstruction. Code, models and results are available at https://nianticlabs.github.io/simplerecon