Urban Radiance Fields

TL;DR

This paper introduces an advanced neural radiance field method tailored for urban outdoor environments, enabling high-quality 3D reconstruction and novel view synthesis from street-level data with multiple sensor modalities.

Contribution

It extends Neural Radiance Fields to incorporate asynchronously captured lidar data, exposure correction, and segmentation supervision, significantly improving urban scene modeling.

Findings

Achieves state-of-the-art 3D surface reconstruction.

Produces higher quality novel views than traditional and neural methods.

Demonstrates effective urban scene modeling with Street View data.

Abstract

The goal of this work is to perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments (e.g., Street View). Given a sequence of posed RGB images and lidar sweeps acquired by cameras and scanners moving through an outdoor scene, we produce a model from which 3D surfaces can be extracted and novel RGB images can be synthesized. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings, with new methods for leveraging asynchronously captured lidar data, for addressing exposure variation between captured images, and for leveraging predicted image segmentations to supervise densities on rays pointing at the sky. Each of these three extensions provides significant performance improvements in experiments on Street View data. Our system produces state-of-the-art 3D surface reconstructions and synthesizes higher quality novel views in comparison to both traditional methods (e.g.~COLMAP) and recent neural representations (e.g.~Mip-NeRF).