Category-Level 3D Non-Rigid Registration from Single-View RGB Images

TL;DR

This paper introduces a CNN-based method for 3D non-rigid registration from single-view RGB images, enabling shape reconstruction without depth data and outperforming traditional methods like CPD.

Contribution

The novel approach infers deformation fields directly from RGB images using learned shape spaces, eliminating the need for depth sensors in 3D registration.

Findings

Outperforms CPD in registration accuracy for tested categories

Works effectively on transparent and shiny objects without depth data

Reconstructs 3D models from single RGB images

Abstract

In this paper, we propose a novel approach to solve the 3D non-rigid registration problem from RGB images using Convolutional Neural Networks (CNNs). Our objective is to find a deformation field (typically used for transferring knowledge between instances, e.g., grasping skills) that warps a given 3D canonical model into a novel instance observed by a single-view RGB image. This is done by training a CNN that infers a deformation field for the visible parts of the canonical model and by employing a learned shape (latent) space for inferring the deformations of the occluded parts. As result of the registration, the observed model is reconstructed. Because our method does not need depth information, it can register objects that are typically hard to perceive with RGB-D sensors, e.g. with transparent or shiny surfaces. Even without depth data, our approach outperforms the Coherent Point Drift (CPD) registration method for the evaluated object categories.