Few-Shot Generalization for Single-Image 3D Reconstruction via Priors

TL;DR

This paper introduces a category-agnostic model for single-image 3D reconstruction that effectively generalizes to new classes using minimal prior data, without retraining, and improves with multiple views.

Contribution

The proposed architecture reframes 3D reconstruction as a refinement of category-specific priors, enabling zero-shot generalization to new classes with minimal data.

Findings

Outperforms category-agnostic baselines in zero-shot reconstruction

Competitive with finetuning-based methods on new categories

Enhances reconstruction quality with multiple views

Abstract

Recent work on single-view 3D reconstruction shows impressive results, but has been restricted to a few fixed categories where extensive training data is available. The problem of generalizing these models to new classes with limited training data is largely open. To address this problem, we present a new model architecture that reframes single-view 3D reconstruction as learnt, category agnostic refinement of a provided, category-specific prior. The provided prior shape for a novel class can be obtained from as few as one 3D shape from this class. Our model can start reconstructing objects from the novel class using this prior without seeing any training image for this class and without any retraining. Our model outperforms category-agnostic baselines and remains competitive with more sophisticated baselines that finetune on the novel categories. Additionally, our network is capable of improving the reconstruction given multiple views despite not being trained on task of multi-view reconstruction.