Rethinking Reprojection: Closing the Loop for Pose-aware ShapeReconstruction from a Single Image

TL;DR

This paper introduces a novel approach for 3D shape and pose reconstruction from a single image that leverages 2D silhouette annotations and emphasizes reprojecting shapes to improve accuracy.

Contribution

It proposes a new task of pose-aware shape reconstruction and develops architectures that reproject predicted shapes onto images using estimated poses, reducing reliance on 3D labels.

Findings

Outperforms existing methods in predicting pose-aware 3D shapes

Utilizes 2D silhouette annotations instead of 3D labels

Demonstrates effectiveness across multiple object categories

Abstract

An emerging problem in computer vision is the reconstruction of 3D shape and pose of an object from a single image. Hitherto, the problem has been addressed through the application of canonical deep learning methods to regress from the image directly to the 3D shape and pose labels. These approaches, however, are problematic from two perspectives. First, they are minimizing the error between 3D shapes and pose labels - with little thought about the nature of this label error when reprojecting the shape back onto the image. Second, they rely on the onerous and ill-posed task of hand labeling natural images with respect to 3D shape and pose. In this paper we define the new task of pose-aware shape reconstruction from a single image, and we advocate that cheaper 2D annotations of objects silhouettes in natural images can be utilized. We design architectures of pose-aware shape reconstruction which re-project the predicted shape back on to the image using the predicted pose. Our evaluation on several object categories demonstrates the superiority of our method for predicting pose-aware 3D shapes from natural images.