Pre-train, Self-train, Distill: A simple recipe for Supersizing 3D Reconstruction

TL;DR

This paper presents a scalable method for learning a unified 3D reconstruction model across hundreds of object categories using segmented images, improving over prior methods and enabling zero-shot inference.

Contribution

It introduces a unified training approach that leverages segmented image collections for 3D reconstruction across many categories, simplifying training and enhancing generalization.

Findings

Learned 3D inference for over 150 categories.

Improved reconstruction quality over category-specific baselines.

Achieved zero-shot inference on unseen categories.

Abstract

Our work learns a unified model for single-view 3D reconstruction of objects from hundreds of semantic categories. As a scalable alternative to direct 3D supervision, our work relies on segmented image collections for learning 3D of generic categories. Unlike prior works that use similar supervision but learn independent category-specific models from scratch, our approach of learning a unified model simplifies the training process while also allowing the model to benefit from the common structure across categories. Using image collections from standard recognition datasets, we show that our approach allows learning 3D inference for over 150 object categories. We evaluate using two datasets and qualitatively and quantitatively show that our unified reconstruction approach improves over prior category-specific reconstruction baselines. Our final 3D reconstruction model is also capable of zero-shot inference on images from unseen object categories and we empirically show that increasing the number of training categories improves the reconstruction quality.