LidaRF: Delving into Lidar for Neural Radiance Field on Street Scenes

TL;DR

This paper introduces LidaRF, a method that leverages Lidar data to enhance neural radiance fields for realistic street scene rendering, addressing challenges like sparse sampling and camera motion.

Contribution

It proposes a novel framework that fuses Lidar-based geometric information with implicit representations, along with occlusion-aware depth supervision and augmented view generation.

Findings

Significantly improves novel view synthesis quality in street scenes.

Effectively utilizes Lidar data to address sparsity and motion challenges.

Enhances realism and accuracy of 3D scene reconstructions.

Abstract

Photorealistic simulation plays a crucial role in applications such as autonomous driving, where advances in neural radiance fields (NeRFs) may allow better scalability through the automatic creation of digital 3D assets. However, reconstruction quality suffers on street scenes due to largely collinear camera motions and sparser samplings at higher speeds. On the other hand, the application often demands rendering from camera views that deviate from the inputs to accurately simulate behaviors like lane changes. In this paper, we propose several insights that allow a better utilization of Lidar data to improve NeRF quality on street scenes. First, our framework learns a geometric scene representation from Lidar, which is fused with the implicit grid-based representation for radiance decoding, thereby supplying stronger geometric information offered by explicit point cloud. Second, we put forth a robust occlusion-aware depth supervision scheme, which allows utilizing densified Lidar points by accumulation. Third, we generate augmented training views from Lidar points for further improvement. Our insights translate to largely improved novel view synthesis under real driving scenes.