BEV-VAE: Multi-view Image Generation with Spatial Consistency for Autonomous Driving

TL;DR

BEV-VAE introduces a novel multi-view image generation framework that ensures spatial consistency and explicit 3D scene modeling for autonomous driving, enabling controllable view synthesis from a unified BEV latent space.

Contribution

It proposes BEV-VAE, a method combining a BEV latent space with a latent diffusion transformer for consistent multi-view scene generation in autonomous driving.

Findings

Strong performance on nuScenes and Argoverse 2 datasets.

Effective 3D consistent reconstruction and generation.

Supports arbitrary view synthesis with optional 3D layouts.

Abstract

Multi-view image generation in autonomous driving demands consistent 3D scene understanding across camera views. Most existing methods treat this problem as a 2D image set generation task, lacking explicit 3D modeling. However, we argue that a structured representation is crucial for scene generation, especially for autonomous driving applications. This paper proposes BEV-VAE for consistent and controllable view synthesis. BEV-VAE first trains a multi-view image variational autoencoder for a compact and unified BEV latent space and then generates the scene with a latent diffusion transformer. BEV-VAE supports arbitrary view generation given camera configurations, and optionally 3D layouts. Experiments on nuScenes and Argoverse 2 (AV2) show strong performance in both 3D consistent reconstruction and generation. The code is available at: https://github.com/Czm369/bev-vae.