StreetSurf: Extending Multi-view Implicit Surface Reconstruction to Street Views

TL;DR

StreetSurf introduces a multi-view implicit surface reconstruction method tailored for street view images, achieving high-quality geometry and appearance results without LiDAR, suitable for autonomous driving datasets.

Contribution

It extends neural surface reconstruction to unbounded street views, using cuboid boundaries, geometric priors, and efficient ray marching, without relying on LiDAR data.

Findings

Achieves state-of-the-art reconstruction quality in geometry and appearance.

Reconstructs in only 1-2 hours on a single RTX3090 GPU.

Enables downstream tasks like ray tracing and LiDAR simulation.

Abstract

We present a novel multi-view implicit surface reconstruction technique, termed StreetSurf, that is readily applicable to street view images in widely-used autonomous driving datasets, such as Waymo-perception sequences, without necessarily requiring LiDAR data. As neural rendering research expands rapidly, its integration into street views has started to draw interests. Existing approaches on street views either mainly focus on novel view synthesis with little exploration of the scene geometry, or rely heavily on dense LiDAR data when investigating reconstruction. Neither of them investigates multi-view implicit surface reconstruction, especially under settings without LiDAR data. Our method extends prior object-centric neural surface reconstruction techniques to address the unique challenges posed by the unbounded street views that are captured with non-object-centric, long and narrow camera trajectories. We delimit the unbounded space into three parts, close-range, distant-view and sky, with aligned cuboid boundaries, and adapt cuboid/hyper-cuboid hash-grids along with road-surface initialization scheme for finer and disentangled representation. To further address the geometric errors arising from textureless regions and insufficient viewing angles, we adopt geometric priors that are estimated using general purpose monocular models. Coupled with our implementation of efficient and fine-grained multi-stage ray marching strategy, we achieve state of the art reconstruction quality in both geometry and appearance within only one to two hours of training time with a single RTX3090 GPU for each street view sequence. Furthermore, we demonstrate that the reconstructed implicit surfaces have rich potential for various downstream tasks, including ray tracing and LiDAR simulation.