Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene
Tai-Yu Pan, Sooyoung Jeon, Mengdi Fan, Jinsu Yoo, Zhenyang Feng, Mark, Campbell, Kilian Q. Weinberger, Bharath Hariharan, Wei-Lun Chao

TL;DR
This paper introduces Transfer Your Perspective (TYP), a diffusion-based method that generates realistic, viewpoint-conditioned driving scenes from ego-car data, facilitating collaborative perception development for autonomous vehicles.
Contribution
It presents the first solution to generate multi-view driving scenes conditioned on real ego-car data, enabling scalable collaborative autonomous driving datasets.
Findings
TYP produces realistic, consistent multi-view driving scenes.
It enables training collaborative perception models with minimal real-world data.
TYP improves downstream autonomous vehicle perception tasks.
Abstract
Self-driving cars relying solely on ego-centric perception face limitations in sensing, often failing to detect occluded, faraway objects. Collaborative autonomous driving (CAV) seems like a promising direction, but collecting data for development is non-trivial. It requires placing multiple sensor-equipped agents in a real-world driving scene, simultaneously! As such, existing datasets are limited in locations and agents. We introduce a novel surrogate to the rescue, which is to generate realistic perception from different viewpoints in a driving scene, conditioned on a real-world sample - the ego-car's sensory data. This surrogate has huge potential: it could potentially turn any ego-car dataset into a collaborative driving one to scale up the development of CAV. We present the very first solution, using a combination of simulated collaborative data and real ego-car data. Our method,…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsComputer Graphics and Visualization Techniques · 3D Surveying and Cultural Heritage
MethodsDiffusion
