# OpenM3D: Open Vocabulary Multi-view Indoor 3D Object Detection without Human Annotations

## TL;DR

OpenM3D is a novel, annotation-free, multi-view indoor 3D object detector that leverages CLIP features and pseudo-boxes, achieving high accuracy and efficiency in open-vocabulary detection.

## Contribution

It introduces a single-stage open-vocabulary 3D detector trained without human annotations, utilizing pseudo-boxes and CLIP features for improved detection.

## Key findings

- Outperforms existing methods in accuracy on ScanNet200 and ARKitScenes.
- Achieves real-time detection at 0.3 seconds per scene.
- Demonstrates superior speed and accuracy compared to two-stage baselines.

## Abstract

Open-vocabulary (OV) 3D object detection is an emerging field, yet its exploration through image-based methods remains limited compared to 3D point cloud-based methods. We introduce OpenM3D, a novel open-vocabulary multi-view indoor 3D object detector trained without human annotations. In particular, OpenM3D is a single-stage detector adapting the 2D-induced voxel features from the ImGeoNet model. To support OV, it is jointly trained with a class-agnostic 3D localization loss requiring high-quality 3D pseudo boxes and a voxel-semantic alignment loss requiring diverse pre-trained CLIP features. We follow the training setting of OV-3DET where posed RGB-D images are given but no human annotations of 3D boxes or classes are available. We propose a 3D Pseudo Box Generation method using a graph embedding technique that combines 2D segments into coherent 3D structures. Our pseudo-boxes achieve higher precision and recall than other methods, including the method proposed in OV-3DET. We further sample diverse CLIP features from 2D segments associated with each coherent 3D structure to align with the corresponding voxel feature. The key to training a highly accurate single-stage detector requires both losses to be learned toward high-quality targets. At inference, OpenM3D, a highly efficient detector, requires only multi-view images for input and demonstrates superior accuracy and speed (0.3 sec. per scene) on ScanNet200 and ARKitScenes indoor benchmarks compared to existing methods. We outperform a strong two-stage method that leverages our class-agnostic detector with a ViT CLIP-based OV classifier and a baseline incorporating multi-view depth estimator on both accuracy and speed.