SSR: Semi-supervised Soft Rasterizer for single-view 2D to 3D Reconstruction

TL;DR

This paper introduces SSR, a semi-supervised method that enhances 3D object reconstruction from 2D images by leveraging unlabeled data through a viewpoint prediction network, reducing labeling effort and improving accuracy.

Contribution

The work proposes a novel semi-supervised approach using a Siamese network for viewpoint estimation, enabling effective use of unlabeled images in 3D reconstruction.

Findings

Significant IoU improvement on ShapeNet with minimal labeled data

Effective viewpoint prediction for unlabeled images

Enhanced 3D reconstruction accuracy using semi-supervised learning

Abstract

Recent work has made significant progress in learning object meshes with weak supervision. Soft Rasterization methods have achieved accurate 3D reconstruction from 2D images with viewpoint supervision only. In this work, we further reduce the labeling effort by allowing such 3D reconstruction methods leverage unlabeled images. In order to obtain the viewpoints for these unlabeled images, we propose to use a Siamese network that takes two images as input and outputs whether they correspond to the same viewpoint. During training, we minimize the cross entropy loss to maximize the probability of predicting whether a pair of images belong to the same viewpoint or not. To get the viewpoint of a new image, we compare it against different viewpoints obtained from the training samples and select the viewpoint with the highest matching probability. We finally label the unlabeled images with the most confident predicted viewpoint and train a deep network that has a differentiable rasterization layer. Our experiments show that even labeling only two objects yields significant improvement in IoU for ShapeNet when leveraging unlabeled examples. Code is available at https://github.com/IssamLaradji/SSR.