G3R: Gradient Guided Generalizable Reconstruction

TL;DR

G3R introduces a gradient-guided, generalizable 3D reconstruction method that efficiently produces high-quality large scene models, combining the advantages of per-scene optimization and fast prediction, with significant speed and robustness improvements.

Contribution

The paper presents G3R, a novel approach that uses gradient feedback to enable fast, high-quality, and generalizable 3D scene reconstruction for large scenes.

Findings

G3R accelerates reconstruction by at least 10x.

G3R achieves comparable or better realism than existing methods.

G3R is more robust to large view changes.

Abstract

Large scale 3D scene reconstruction is important for applications such as virtual reality and simulation. Existing neural rendering approaches (e.g., NeRF, 3DGS) have achieved realistic reconstructions on large scenes, but optimize per scene, which is expensive and slow, and exhibit noticeable artifacts under large view changes due to overfitting. Generalizable approaches or large reconstruction models are fast, but primarily work for small scenes/objects and often produce lower quality rendering results. In this work, we introduce G3R, a generalizable reconstruction approach that can efficiently predict high-quality 3D scene representations for large scenes. We propose to learn a reconstruction network that takes the gradient feedback signals from differentiable rendering to iteratively update a 3D scene representation, combining the benefits of high photorealism from per-scene optimization with data-driven priors from fast feed-forward prediction methods. Experiments on urban-driving and drone datasets show that G3R generalizes across diverse large scenes and accelerates the reconstruction process by at least 10x while achieving comparable or better realism compared to 3DGS, and also being more robust to large view changes.