Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene

TL;DR

This paper introduces Transfer Your Perspective (TYP), a diffusion-based method that generates realistic, viewpoint-conditioned driving scenes from ego-car data, facilitating collaborative perception development for autonomous vehicles.

Contribution

It presents the first solution to generate multi-view driving scenes conditioned on real ego-car data, enabling scalable collaborative autonomous driving datasets.

Findings

TYP produces realistic, consistent multi-view driving scenes.

It enables training collaborative perception models with minimal real-world data.

TYP improves downstream autonomous vehicle perception tasks.

Abstract

Self-driving cars relying solely on ego-centric perception face limitations in sensing, often failing to detect occluded, faraway objects. Collaborative autonomous driving (CAV) seems like a promising direction, but collecting data for development is non-trivial. It requires placing multiple sensor-equipped agents in a real-world driving scene, simultaneously! As such, existing datasets are limited in locations and agents. We introduce a novel surrogate to the rescue, which is to generate realistic perception from different viewpoints in a driving scene, conditioned on a real-world sample - the ego-car's sensory data. This surrogate has huge potential: it could potentially turn any ego-car dataset into a collaborative driving one to scale up the development of CAV. We present the very first solution, using a combination of simulated collaborative data and real ego-car data. Our method, Transfer Your Perspective (TYP), learns a conditioned diffusion model whose output samples are not only realistic but also consistent in both semantics and layouts with the given ego-car data. Empirical results demonstrate TYP's effectiveness in aiding in a CAV setting. In particular, TYP enables us to (pre-)train collaborative perception algorithms like early and late fusion with little or no real-world collaborative data, greatly facilitating downstream CAV applications.