Multi-Category Mesh Reconstruction From Image Collections

TL;DR

This paper introduces a method for reconstructing textured 3D meshes from images across multiple categories without relying on category-specific templates, using only foreground masks and rough camera poses for supervision.

Contribution

It presents a novel approach that learns category-level deformable models and predicts instance-specific deformations, enabling multi-category 3D reconstruction without category-specific priors.

Findings

Achieves state-of-the-art results on public datasets.

Effectively distinguishes between object categories.

Produces smooth, high-quality 3D meshes.

Abstract

Recently, learning frameworks have shown the capability of inferring the accurate shape, pose, and texture of an object from a single RGB image. However, current methods are trained on image collections of a single category in order to exploit specific priors, and they often make use of category-specific 3D templates. In this paper, we present an alternative approach that infers the textured mesh of objects combining a series of deformable 3D models and a set of instance-specific deformation, pose, and texture. Differently from previous works, our method is trained with images of multiple object categories using only foreground masks and rough camera poses as supervision. Without specific 3D templates, the framework learns category-level models which are deformed to recover the 3D shape of the depicted object. The instance-specific deformations are predicted independently for each vertex of the learned 3D mesh, enabling the dynamic subdivision of the mesh during the training process. Experiments show that the proposed framework can distinguish between different object categories and learn category-specific shape priors in an unsupervised manner. Predicted shapes are smooth and can leverage from multiple steps of subdivision during the training process, obtaining comparable or state-of-the-art results on two public datasets. Models and code are publicly released.