A Coarse-to-Fine Approach to Multi-Modality 3D Occupancy Grounding

TL;DR

This paper introduces a new benchmark and an end-to-end model for 3D occupancy grounding in outdoor scenes, improving object perception accuracy by integrating multi-modal features and a coarse-to-fine approach.

Contribution

It presents a novel benchmark based on nuScenes for voxel-level 3D occupancy grounding and proposes GroundingOcc, a multi-modal model that enhances localization and occupancy prediction.

Findings

Outperforms existing baselines on the 3D occupancy grounding benchmark.

Integrates visual, textual, and point cloud data for improved accuracy.

Utilizes a coarse-to-fine approach with additional modules for geometric understanding.

Abstract

Visual grounding aims to identify objects or regions in a scene based on natural language descriptions, essential for spatially aware perception in autonomous driving. However, existing visual grounding tasks typically depend on bounding boxes that often fail to capture fine-grained details. Not all voxels within a bounding box are occupied, resulting in inaccurate object representations. To address this, we introduce a benchmark for 3D occupancy grounding in challenging outdoor scenes. Built on the nuScenes dataset, it integrates natural language with voxel-level occupancy annotations, offering more precise object perception compared to the traditional grounding task. Moreover, we propose GroundingOcc, an end-to-end model designed for 3D occupancy grounding through multi-modal learning. It combines visual, textual, and point cloud features to predict object location and occupancy information from coarse to fine. Specifically, GroundingOcc comprises a multimodal encoder for feature extraction, an occupancy head for voxel-wise predictions, and a grounding head to refine localization. Additionally, a 2D grounding module and a depth estimation module enhance geometric understanding, thereby boosting model performance. Extensive experiments on the benchmark demonstrate that our method outperforms existing baselines on 3D occupancy grounding. The dataset is available at https://github.com/RONINGOD/GroundingOcc.