Efficient Depth-Guided Urban View Synthesis

TL;DR

EDUS is a novel method for fast, efficient urban view synthesis that leverages noisy geometric priors for robust, generalizable performance from sparse images, outperforming prior methods in speed and accuracy.

Contribution

Introduces EDUS, a new approach that uses noisy geometric priors for efficient, generalizable street view synthesis from sparse images with fast inference and fine-tuning.

Findings

State-of-the-art performance in sparse view synthesis.

Robust generalization across diverse street scenes.

Fast inference and fine-tuning capabilities.

Abstract

Recent advances in implicit scene representation enable high-fidelity street view novel view synthesis. However, existing methods optimize a neural radiance field for each scene, relying heavily on dense training images and extensive computation resources. To mitigate this shortcoming, we introduce a new method called Efficient Depth-Guided Urban View Synthesis (EDUS) for fast feed-forward inference and efficient per-scene fine-tuning. Different from prior generalizable methods that infer geometry based on feature matching, EDUS leverages noisy predicted geometric priors as guidance to enable generalizable urban view synthesis from sparse input images. The geometric priors allow us to apply our generalizable model directly in the 3D space, gaining robustness across various sparsity levels. Through comprehensive experiments on the KITTI-360 and Waymo datasets, we demonstrate promising generalization abilities on novel street scenes. Moreover, our results indicate that EDUS achieves state-of-the-art performance in sparse view settings when combined with fast test-time optimization.