DreamCar: Leveraging Car-specific Prior for in-the-wild 3D Car Reconstruction

TL;DR

DreamCar introduces a novel approach for high-quality 3D car reconstruction from limited in-the-wild images by leveraging a car-specific generative prior, symmetry, and pose optimization, outperforming existing methods.

Contribution

The paper presents DreamCar, a new method that uses a car-specific generative model, symmetry, and pose correction to improve 3D car reconstruction from few images.

Findings

Outperforms existing methods in 3D car reconstruction quality.

Uses a new Car360 dataset to enhance generative model robustness.

Effectively reconstructs 3D cars from single or few images.

Abstract

Self-driving industries usually employ professional artists to build exquisite 3D cars. However, it is expensive to craft large-scale digital assets. Since there are already numerous datasets available that contain a vast number of images of cars, we focus on reconstructing high-quality 3D car models from these datasets. However, these datasets only contain one side of cars in the forward-moving scene. We try to use the existing generative models to provide more supervision information, but they struggle to generalize well in cars since they are trained on synthetic datasets not car-specific. In addition, The reconstructed 3D car texture misaligns due to a large error in camera pose estimation when dealing with in-the-wild images. These restrictions make it challenging for previous methods to reconstruct complete 3D cars. To address these problems, we propose a novel method, named DreamCar, which can reconstruct high-quality 3D cars given a few images even a single image. To generalize the generative model, we collect a car dataset, named Car360, with over 5,600 vehicles. With this dataset, we make the generative model more robust to cars. We use this generative prior specific to the car to guide its reconstruction via Score Distillation Sampling. To further complement the supervision information, we utilize the geometric and appearance symmetry of cars. Finally, we propose a pose optimization method that rectifies poses to tackle texture misalignment. Extensive experiments demonstrate that our method significantly outperforms existing methods in reconstructing high-quality 3D cars. \href{https://xiaobiaodu.github.io/dreamcar-project/}{Our code is available.}