Weakly supervised 3D Reconstruction with Adversarial Constraint

TL;DR

This paper introduces a weakly supervised 3D reconstruction method using 2D masks and adversarial constraints, reducing the need for expensive 3D annotations while maintaining high reconstruction quality.

Contribution

It proposes a novel approach combining weak supervision with adversarial constraints to improve 3D reconstruction from 2D masks.

Findings

Effective reconstruction with minimal supervision

Comparable results to fully supervised methods

Applicable to both synthetic and real images

Abstract

Supervised 3D reconstruction has witnessed a significant progress through the use of deep neural networks. However, this increase in performance requires large scale annotations of 2D/3D data. In this paper, we explore inexpensive 2D supervision as an alternative for expensive 3D CAD annotation. Specifically, we use foreground masks as weak supervision through a raytrace pooling layer that enables perspective projection and backpropagation. Additionally, since the 3D reconstruction from masks is an ill posed problem, we propose to constrain the 3D reconstruction to the manifold of unlabeled realistic 3D shapes that match mask observations. We demonstrate that learning a log-barrier solution to this constrained optimization problem resembles the GAN objective, enabling the use of existing tools for training GANs. We evaluate and analyze the manifold constrained reconstruction on various datasets for single and multi-view reconstruction of both synthetic and real images.