Weakly-Supervised End-to-End CAD Retrieval to Scan Objects

TL;DR

This paper introduces a weakly-supervised method for retrieving CAD models from 3D scans without needing explicit CAD-scan associations, using only object detection info, and outperforms supervised methods especially in zero-shot scenarios.

Contribution

It presents a novel end-to-end weakly-supervised CAD retrieval approach that leverages a differentiable top-k layer and requires only bounding box annotations.

Findings

Outperforms fully-supervised methods on ScanNet scans.

Maintains robustness for unseen object categories.

Achieves significant improvements in zero-shot CAD retrieval.

Abstract

CAD model retrieval to real-world scene observations has shown strong promise as a basis for 3D perception of objects and a clean, lightweight mesh-based scene representation; however, current approaches to retrieve CAD models to a query scan rely on expensive manual annotations of 1:1 associations of CAD-scan objects, which typically contain strong lower-level geometric differences. We thus propose a new weakly-supervised approach to retrieve semantically and structurally similar CAD models to a query 3D scanned scene without requiring any CAD-scan associations, and only object detection information as oriented bounding boxes. Our approach leverages a fully-differentiable top-$k$ retrieval layer, enabling end-to-end training guided by geometric and perceptual similarity of the top retrieved CAD models to the scan queries. We demonstrate that our weakly-supervised approach can outperform fully-supervised retrieval methods on challenging real-world ScanNet scans, and maintain robustness for unseen class categories, achieving significantly improved performance over fully-supervised state of the art in zero-shot CAD retrieval.