3D-COCO: extension of MS-COCO dataset for image detection and 3D reconstruction modules

TL;DR

3D-COCO extends the MS-COCO dataset by adding 3D models and alignment annotations, enabling advanced 3D reconstruction and detection research using combined 2D and 3D data.

Contribution

It introduces a large-scale, open-source dataset with 3D models and 2D-3D alignments, facilitating research in 3D computer vision tasks.

Findings

28K 3D models added from ShapeNet and Objaverse

Effective IoU-based matching for 2D-3D alignment

Open-source dataset available for research

Abstract

We introduce 3D-COCO, an extension of the original MS-COCO dataset providing 3D models and 2D-3D alignment annotations. 3D-COCO was designed to achieve computer vision tasks such as 3D reconstruction or image detection configurable with textual, 2D image, and 3D CAD model queries. We complete the existing MS-COCO dataset with 28K 3D models collected on ShapeNet and Objaverse. By using an IoU-based method, we match each MS-COCO annotation with the best 3D models to provide a 2D-3D alignment. The open-source nature of 3D-COCO is a premiere that should pave the way for new research on 3D-related topics. The dataset and its source codes is available at https://kalisteo.cea.fr/index.php/coco3d-object-detection-and-reconstruction/