Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve

TL;DR

Mask2CAD introduces a method that detects objects in images, retrieves the most similar 3D CAD models and poses, creating a lightweight, accurate 3D representation that bridges real-world images and synthetic models.

Contribution

The paper presents a novel joint embedding space for image regions and CAD models, enabling effective retrieval and pose estimation for 3D shape prediction from images.

Findings

Outperforms state-of-the-art on Pix3D dataset

Provides a new image-to-3D baseline with diverse shapes and occlusions

Demonstrates effective real-world 3D shape retrieval and pose estimation

Abstract

Object recognition has seen significant progress in the image domain, with focus primarily on 2D perception. We propose to leverage existing large-scale datasets of 3D models to understand the underlying 3D structure of objects seen in an image by constructing a CAD-based representation of the objects and their poses. We present Mask2CAD, which jointly detects objects in real-world images and for each detected object, optimizes for the most similar CAD model and its pose. We construct a joint embedding space between the detected regions of an image corresponding to an object and 3D CAD models, enabling retrieval of CAD models for an input RGB image. This produces a clean, lightweight representation of the objects in an image; this CAD-based representation ensures a valid, efficient shape representation for applications such as content creation or interactive scenarios, and makes a step towards understanding the transformation of real-world imagery to a synthetic domain. Experiments on real-world images from Pix3D demonstrate the advantage of our approach in comparison to state of the art. To facilitate future research, we additionally propose a new image-to-3D baseline on ScanNet which features larger shape diversity, real-world occlusions, and challenging image views.