OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding

TL;DR

OpenVoxel is a training-free method that groups and captions voxels in 3D scenes using vision-language models, enabling open-vocabulary scene understanding without additional training or embeddings.

Contribution

It introduces a novel training-free approach for voxel grouping and captioning in 3D scenes, leveraging existing vision-language models for open-vocabulary tasks.

Findings

Outperforms recent methods in complex referring expression segmentation

Does not require training or embeddings from CLIP/BERT

Effectively builds scene maps with meaningful object groups

Abstract

We propose OpenVoxel, a training-free algorithm for grouping and captioning sparse voxels for the open-vocabulary 3D scene understanding tasks. Given the sparse voxel rasterization (SVR) model obtained from multi-view images of a 3D scene, our OpenVoxel is able to produce meaningful groups that describe different objects in the scene. Also, by leveraging powerful Vision Language Models (VLMs) and Multi-modal Large Language Models (MLLMs), our OpenVoxel successfully build an informative scene map by captioning each group, enabling further 3D scene understanding tasks such as open-vocabulary segmentation (OVS) or referring expression segmentation (RES). Unlike previous methods, our method is training-free and does not introduce embeddings from a CLIP/BERT text encoder. Instead, we directly proceed with text-to-text search using MLLMs. Through extensive experiments, our method demonstrates superior performance compared to recent studies, particularly in complex referring expression segmentation (RES) tasks. The code will be open.