Drive-1-to-3: Enriching Diffusion Priors for Novel View Synthesis of Real Vehicles

TL;DR

This paper presents a set of practical strategies for fine-tuning large diffusion models to improve novel view synthesis of real vehicles, addressing synthetic-to-real domain gaps for autonomous driving applications.

Contribution

It introduces specific techniques like geometric alignment, object-centric data curation, occlusion-aware training, and symmetry priors to enhance model performance on real-world vehicle images.

Findings

Achieved a 68.8% reduction in FID score for real vehicle view synthesis.

Developed methods to handle occlusions and varying object scales effectively.

Enhanced the applicability of diffusion models to real-world autonomous driving data.

Abstract

The recent advent of large-scale 3D data, e.g. Objaverse, has led to impressive progress in training pose-conditioned diffusion models for novel view synthesis. However, due to the synthetic nature of such 3D data, their performance drops significantly when applied to real-world images. This paper consolidates a set of good practices to finetune large pretrained models for a real-world task -- harvesting vehicle assets for autonomous driving applications. To this end, we delve into the discrepancies between the synthetic data and real driving data, then develop several strategies to account for them properly. Specifically, we start with a virtual camera rotation of real images to ensure geometric alignment with synthetic data and consistency with the pose manifold defined by pretrained models. We also identify important design choices in object-centric data curation to account for varying object distances in real driving scenes -- learn across varying object scales with fixed camera focal length. Further, we perform occlusion-aware training in latent spaces to account for ubiquitous occlusions in real data, and handle large viewpoint changes by leveraging a symmetric prior. Our insights lead to effective finetuning that results in a $68.8\%$ reduction in FID for novel view synthesis over prior arts.