ROCA: Robust CAD Model Retrieval and Alignment from a Single Image

TL;DR

ROCA is an end-to-end method that retrieves and aligns 3D CAD models from a single image, enabling 3D scene understanding with improved accuracy through dense correspondences and differentiable optimization.

Contribution

It introduces a novel differentiable alignment optimization and a unified approach for CAD retrieval and alignment from a single RGB image.

Findings

Significantly improves CAD alignment accuracy on real-world data.

Leverages dense 2D-3D correspondences for robust alignment.

Achieves state-of-the-art performance, increasing accuracy from 9.5% to 17.6%.

Abstract

We present ROCA, a novel end-to-end approach that retrieves and aligns 3D CAD models from a shape database to a single input image. This enables 3D perception of an observed scene from a 2D RGB observation, characterized as a lightweight, compact, clean CAD representation. Core to our approach is our differentiable alignment optimization based on dense 2D-3D object correspondences and Procrustes alignment. ROCA can thus provide a robust CAD alignment while simultaneously informing CAD retrieval by leveraging the 2D-3D correspondences to learn geometrically similar CAD models. Experiments on challenging, real-world imagery from ScanNet show that ROCA significantly improves on state of the art, from 9.5% to 17.6% in retrieval-aware CAD alignment accuracy.