HQ-OV3D: A High Box Quality Open-World 3D Detection Framework based on Diffision Model

TL;DR

HQ-OV3D introduces a novel open-world 3D detection framework that enhances pseudo-label quality using cross-modality geometric consistency and denoising, significantly improving detection accuracy for novel classes.

Contribution

The paper presents a new framework combining cross-modality proposals and denoising to generate high-quality pseudo-labels for open-vocabulary 3D detection.

Findings

7.37% mAP improvement on novel classes

Superior pseudo-label quality over state-of-the-art methods

Effective as a standalone detector and pseudo-label generator

Abstract

Traditional closed-set 3D detection frameworks fail to meet the demands of open-world applications like autonomous driving. Existing open-vocabulary 3D detection methods typically adopt a two-stage pipeline consisting of pseudo-label generation followed by semantic alignment. While vision-language models (VLMs) recently have dramatically improved the semantic accuracy of pseudo-labels, their geometric quality, particularly bounding box precision, remains commonly neglected. To address this issue, we propose a High Box Quality Open-Vocabulary 3D Detection (HQ-OV3D) framework, dedicated to generate and refine high-quality pseudo-labels for open-vocabulary classes. The framework comprises two key components: an Intra-Modality Cross-Validated (IMCV) Proposal Generator that utilizes cross-modality geometric consistency to generate high-quality initial 3D proposals, and an Annotated-Class Assisted (ACA) Denoiser that progressively refines 3D proposals by leveraging geometric priors from annotated categories through a DDIM-based denoising mechanism. Compared to the state-of-the-art method, training with pseudo-labels generated by our approach achieves a 7.37% improvement in mAP on novel classes, demonstrating the superior quality of the pseudo-labels produced by our framework. HQ-OV3D can serve not only as a strong standalone open-vocabulary 3D detector but also as a plug-in high-quality pseudo-label generator for existing open-vocabulary detection or annotation pipelines.