Leveraging Geometry for Shape Estimation from a Single RGB Image

TL;DR

This paper introduces a geometry-aware method for 3D shape and pose estimation from a single RGB image, utilizing cross-domain keypoint matches to improve accuracy and enable shape adaptation, outperforming previous approaches.

Contribution

It presents a novel approach that leverages geometric keypoint matching to enhance pose prediction and allows shape modification, addressing limitations of CAD model retrieval methods.

Findings

Improved AP mesh scores on Pix3D dataset for seen objects from 33.2 to 37.8.

Enhanced shape prediction accuracy for unseen objects from 8.2 to 17.1.

Achieved more accurate shape predictions without relying solely on CAD model matching.

Abstract

Predicting 3D shapes and poses of static objects from a single RGB image is an important research area in modern computer vision. Its applications range from augmented reality to robotics and digital content creation. Typically this task is performed through direct object shape and pose predictions which is inaccurate. A promising research direction ensures meaningful shape predictions by retrieving CAD models from large scale databases and aligning them to the objects observed in the image. However, existing work does not take the object geometry into account, leading to inaccurate object pose predictions, especially for unseen objects. In this work we demonstrate how cross-domain keypoint matches from an RGB image to a rendered CAD model allow for more precise object pose predictions compared to ones obtained through direct predictions. We further show that keypoint matches can not only be used to estimate the pose of an object, but also to modify the shape of the object itself. This is important as the accuracy that can be achieved with object retrieval alone is inherently limited to the available CAD models. Allowing shape adaptation bridges the gap between the retrieved CAD model and the observed shape. We demonstrate our approach on the challenging Pix3D dataset. The proposed geometric shape prediction improves the AP mesh over the state-of-the-art from 33.2 to 37.8 on seen objects and from 8.2 to 17.1 on unseen objects. Furthermore, we demonstrate more accurate shape predictions without closely matching CAD models when following the proposed shape adaptation. Code is publicly available at https://github.com/florianlanger/leveraging_geometry_for_shape_estimation .