ODAM: Object Detection, Association, and Mapping using Posed RGB Video

TL;DR

ODAM is a system that combines deep learning and graph neural networks to detect, associate, and map 3D objects from RGB videos, advancing scene understanding for AR and robotics.

Contribution

It introduces a novel pipeline integrating 3D object detection, association, and mapping using posed RGB videos with GNNs and super-quadrics for improved accuracy.

Findings

Significant improvement over existing RGB-only methods on ScanNet.

Effective multi-view geometry optimization of object volumes.

Robust object association across frames.

Abstract

Localizing objects and estimating their extent in 3D is an important step towards high-level 3D scene understanding, which has many applications in Augmented Reality and Robotics. We present ODAM, a system for 3D Object Detection, Association, and Mapping using posed RGB videos. The proposed system relies on a deep learning front-end to detect 3D objects from a given RGB frame and associate them to a global object-based map using a graph neural network (GNN). Based on these frame-to-model associations, our back-end optimizes object bounding volumes, represented as super-quadrics, under multi-view geometry constraints and the object scale prior. We validate the proposed system on ScanNet where we show a significant improvement over existing RGB-only methods.