SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior
Zhongrui Yu, Haoran Wang, Jinze Yang, Hanzhang Wang, Zeke Xie, Yunfeng, Cai, Jiale Cao, Zhong Ji, and Mingming Sun

TL;DR
This paper introduces a novel street view synthesis method that combines 3D Gaussian Splatting with a diffusion prior and multi-modal data to improve rendering quality at novel viewpoints in autonomous driving scenarios.
Contribution
It enhances 3D Gaussian Splatting by integrating a diffusion model and multi-modal data, enabling better generalization to unseen viewpoints in street scenes.
Findings
Outperforms current state-of-the-art models in street view synthesis.
Improves rendering quality at viewpoints deviating significantly from training views.
Leverages LiDAR and adjacent frame images to enhance spatial and contextual understanding.
Abstract
Novel View Synthesis (NVS) for street scenes play a critical role in the autonomous driving simulation. The current mainstream technique to achieve it is neural rendering, such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS). Although thrilling progress has been made, when handling street scenes, current methods struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. To tackle this problem, we propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model along with complementary multi-modal data. Specifically, we first fine-tune a Diffusion Model by adding images from adjacent frames as condition, meanwhile exploiting depth data from LiDAR point clouds to supply additional…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsVideo Surveillance and Tracking Methods · Remote Sensing and LiDAR Applications · Advanced Vision and Imaging
MethodsDiffusion
