ArbiViewGen: Controllable Arbitrary Viewpoint Camera Data Generation for Autonomous Driving via Stable Diffusion Models

TL;DR

ArbiViewGen introduces a diffusion-based framework that enables controllable generation of arbitrary viewpoint images for autonomous driving, overcoming the lack of ground-truth data through innovative self-supervised learning and view stitching techniques.

Contribution

The paper presents ArbiViewGen, a novel controllable image generation method for autonomous driving that does not require ground-truth extrapolated views, using feature-aware stitching and self-supervised learning.

Findings

First controllable arbitrary view generation method for autonomous driving.

Effective self-supervised training without ground-truth extrapolated data.

High-quality, multi-view consistent images from multi-camera setups.

Abstract

Arbitrary viewpoint image generation holds significant potential for autonomous driving, yet remains a challenging task due to the lack of ground-truth data for extrapolated views, which hampers the training of high-fidelity generative models. In this work, we propose Arbiviewgen, a novel diffusion-based framework for the generation of controllable camera images from arbitrary points of view. To address the absence of ground-truth data in unseen views, we introduce two key components: Feature-Aware Adaptive View Stitching (FAVS) and Cross-View Consistency Self-Supervised Learning (CVC-SSL). FAVS employs a hierarchical matching strategy that first establishes coarse geometric correspondences using camera poses, then performs fine-grained alignment through improved feature matching algorithms, and identifies high-confidence matching regions via clustering analysis. Building upon this, CVC-SSL adopts a self-supervised training paradigm where the model reconstructs the original camera views from the synthesized stitched images using a diffusion model, enforcing cross-view consistency without requiring supervision from extrapolated data. Our framework requires only multi-camera images and their associated poses for training, eliminating the need for additional sensors or depth maps. To our knowledge, Arbiviewgen is the first method capable of controllable arbitrary view camera image generation in multiple vehicle configurations.