Share With Thy Neighbors: Single-View Reconstruction by Cross-Instance Consistency

TL;DR

This paper introduces UNICORN, a novel method for single-view 3D reconstruction that leverages cross-instance consistency without requiring traditional supervision like viewpoints or silhouettes, enabling learning from unlabelled image collections.

Contribution

The paper proposes two innovative strategies—progressive conditioning and neighbor reconstruction—for leveraging cross-instance consistency in unlabelled images, along with a structured autoencoding architecture and an adapted differential rendering formulation.

Findings

Effective on synthetic ShapeNet dataset

Performs well on real-image benchmarks like Pascal3D+ Car and CUB

Applicable to real-world collections without silhouettes or cropping

Abstract

Approaches for single-view reconstruction typically rely on viewpoint annotations, silhouettes, the absence of background, multiple views of the same instance, a template shape, or symmetry. We avoid all such supervision and assumptions by explicitly leveraging the consistency between images of different object instances. As a result, our method can learn from large collections of unlabelled images depicting the same object category. Our main contributions are two ways for leveraging cross-instance consistency: (i) progressive conditioning, a training strategy to gradually specialize the model from category to instances in a curriculum learning fashion; and (ii) neighbor reconstruction, a loss enforcing consistency between instances having similar shape or texture. Also critical to the success of our method are: our structured autoencoding architecture decomposing an image into explicit shape, texture, pose, and background; an adapted formulation of differential rendering; and a new optimization scheme alternating between 3D and pose learning. We compare our approach, UNICORN, both on the diverse synthetic ShapeNet dataset - the classical benchmark for methods requiring multiple views as supervision - and on standard real-image benchmarks (Pascal3D+ Car, CUB) for which most methods require known templates and silhouette annotations. We also showcase applicability to more challenging real-world collections (CompCars, LSUN), where silhouettes are not available and images are not cropped around the object.