Learning Category-Specific Mesh Reconstruction from Image Collections

TL;DR

This paper introduces a learning framework that reconstructs 3D object shapes, textures, and keypoints from single images using category-specific deformable meshes trained on annotated image collections without requiring ground-truth 3D data.

Contribution

It proposes a novel deformable mesh-based 3D reconstruction method that learns from 2D image collections, enabling texture inference and keypoint association without 3D supervision.

Findings

Achieves accurate 3D shape and texture prediction on CUB and PASCAL3D datasets.

Enables semantic keypoint association with predicted shapes.

Operates without ground-truth 3D or multi-view supervision.

Abstract

We present a learning framework for recovering the 3D shape, camera, and texture of an object from a single image. The shape is represented as a deformable 3D mesh model of an object category where a shape is parameterized by a learned mean shape and per-instance predicted deformation. Our approach allows leveraging an annotated image collection for training, where the deformable model and the 3D prediction mechanism are learned without relying on ground-truth 3D or multi-view supervision. Our representation enables us to go beyond existing 3D prediction approaches by incorporating texture inference as prediction of an image in a canonical appearance space. Additionally, we show that semantic keypoints can be easily associated with the predicted shapes. We present qualitative and quantitative results of our approach on CUB and PASCAL3D datasets and show that we can learn to predict diverse shapes and textures across objects using only annotated image collections. The project website can be found at https://akanazawa.github.io/cmr/.