Part-level Car Parsing and Reconstruction from Single Street View

TL;DR

This paper introduces a novel framework for 3D car shape, pose, and part estimation from single street view images, leveraging synthesized data and implicit feature transfer, achieving state-of-the-art results.

Contribution

It presents the first unified approach for 3D car reconstruction and pose estimation from street views using part features and a new high-quality dataset.

Findings

Significant improvement in part segmentation accuracy after implicit transfer.

State-of-the-art performance on ApolloCar3D dataset.

Outperforms existing methods by over 8 percentage points in mean A3DP-Abs.

Abstract

Part information has been shown to be resistant to occlusions and viewpoint changes, which is beneficial for various vision-related tasks. However, we found very limited work in car pose estimation and reconstruction from street views leveraging the part information. There are two major contributions in this paper. Firstly, we make the first attempt to build a framework to simultaneously estimate shape, translation, orientation, and semantic parts of cars in 3D space from a single street view. As it is labor-intensive to annotate semantic parts on real street views, we propose a specific approach to implicitly transfer part features from synthesized images to real street views. For pose and shape estimation, we propose a novel network structure that utilizes both part features and 3D losses. Secondly, we are the first to construct a high-quality dataset that contains 348 different car models with physical dimensions and part-level annotations based on global and local deformations. Given these models, we further generate 60K synthesized images with randomization of orientation, illumination, occlusion, and texture. Our results demonstrate that our part segmentation performance is significantly improved after applying our implicit transfer approach. Our network for pose and shape estimation achieves the state-of-the-art performance on the ApolloCar3D dataset and outperforms 3D-RCNN and DeepMANTA by 12.57 and 8.91 percentage points in terms of mean A3DP-Abs.