Towards 3D Objectness Learning in an Open World

TL;DR

This paper introduces OP3Det, a novel 3D object detector capable of recognizing all objects in a scene, including unseen ones, by leveraging 2D and 3D priors and cross-modal learning, significantly outperforming existing methods.

Contribution

The paper proposes OP3Det, a class-agnostic open-world 3D detector that does not rely on text prompts, integrating 2D foundation models and cross-modal features for generalized 3D objectness detection.

Findings

OP3Det surpasses existing open-world 3D detectors by up to 16.0% in AR.

OP3Det achieves a 13.5% improvement over closed-world 3D detectors.

Extensive experiments validate the effectiveness of OP3Det in open-world scenarios.

Abstract

Recent advancements in 3D object detection and novel category detection have made significant progress, yet research on learning generalized 3D objectness remains insufficient. In this paper, we delve into learning open-world 3D objectness, which focuses on detecting all objects in a 3D scene, including novel objects unseen during training. Traditional closed-set 3D detectors struggle to generalize to open-world scenarios, while directly incorporating 3D open-vocabulary models for open-world ability struggles with vocabulary expansion and semantic overlap. To achieve generalized 3D object discovery, We propose OP3Det, a class-agnostic Open-World Prompt-free 3D Detector to detect any objects within 3D scenes without relying on hand-crafted text prompts. We introduce the strong generalization and zero-shot capabilities of 2D foundation models, utilizing both 2D semantic priors and 3D geometric priors for class-agnostic proposals to broaden 3D object discovery. Then, by integrating complementary information from point cloud and RGB image in the cross-modal mixture of experts, OP3Det dynamically routes uni-modal and multi-modal features to learn generalized 3D objectness. Extensive experiments demonstrate the extraordinary performance of OP3Det, which significantly surpasses existing open-world 3D detectors by up to 16.0% in AR and achieves a 13.5% improvement compared to closed-world 3D detectors.