Zoo3D: Zero-Shot 3D Object Detection at Scene Level

TL;DR

Zoo3D introduces a training-free, zero-shot 3D object detection framework that constructs 3D bounding boxes and assigns semantic labels without prior training, achieving state-of-the-art results on multiple benchmarks.

Contribution

It is the first training-free 3D detection method that operates in zero-shot mode and extends to images, significantly advancing open-vocabulary 3D understanding.

Findings

Zero-shot Zoo3D$_0$ outperforms existing self-supervised methods.

Both Zoo3D$_0$ and Zoo3D$_1$ achieve state-of-the-art results.

The method works directly with posed and unposed images.

Abstract

3D object detection is fundamental for spatial understanding. Real-world environments demand models capable of recognizing diverse, previously unseen objects, which remains a major limitation of closed-set methods. Existing open-vocabulary 3D detectors relax annotation requirements but still depend on training scenes, either as point clouds or images. We take this a step further by introducing Zoo3D, the first training-free 3D object detection framework. Our method constructs 3D bounding boxes via graph clustering of 2D instance masks, then assigns semantic labels using a novel open-vocabulary module with best-view selection and view-consensus mask generation. Zoo3D operates in two modes: the zero-shot Zoo3D$_0$, which requires no training at all, and the self-supervised Zoo3D$_1$, which refines 3D box prediction by training a class-agnostic detector on Zoo3D$_0$-generated pseudo labels. Furthermore, we extend Zoo3D beyond point clouds to work directly with posed and even unposed images. Across ScanNet200 and ARKitScenes benchmarks, both Zoo3D$_0$ and Zoo3D$_1$ achieve state-of-the-art results in open-vocabulary 3D object detection. Remarkably, our zero-shot Zoo3D$_0$ outperforms all existing self-supervised methods, hence demonstrating the power and adaptability of training-free, off-the-shelf approaches for real-world 3D understanding. Code is available at https://github.com/col14m/zoo3d .