Beyond Bare Queries: Open-Vocabulary Object Grounding with 3D Scene Graph

TL;DR

This paper introduces BBQ, a modular approach for 3D object grounding in complex natural language queries, utilizing scene graphs, spatial relations, and large language models to improve accuracy and speed in robotics applications.

Contribution

We propose BBQ, a novel modular framework that integrates scene graph reasoning, spatial relations, and large language models for open-vocabulary 3D object grounding.

Findings

BBQ outperforms existing zero-shot methods on Replica and ScanNet datasets.

Leveraging spatial relations improves grounding accuracy in scenes with similar objects.

Our approach achieves real-time performance suitable for robotic applications.

Abstract

Locating objects described in natural language presents a significant challenge for autonomous agents. Existing CLIP-based open-vocabulary methods successfully perform 3D object grounding with simple (bare) queries, but cannot cope with ambiguous descriptions that demand an understanding of object relations. To tackle this problem, we propose a modular approach called BBQ (Beyond Bare Queries), which constructs 3D scene graph representation with metric and semantic spatial edges and utilizes a large language model as a human-to-agent interface through our deductive scene reasoning algorithm. BBQ employs robust DINO-powered associations to construct 3D object-centric map and an advanced raycasting algorithm with a 2D vision-language model to describe them as graph nodes. On the Replica and ScanNet datasets, we have demonstrated that BBQ takes a leading place in open-vocabulary 3D semantic segmentation compared to other zero-shot methods. Also, we show that leveraging spatial relations is especially effective for scenes containing multiple entities of the same semantic class. On challenging Sr3D+, Nr3D and ScanRefer benchmarks, our deductive approach demonstrates a significant improvement, enabling objects grounding by complex queries compared to other state-of-the-art methods. The combination of our design choices and software implementation has resulted in significant data processing speed in experiments on the robot on-board computer. This promising performance enables the application of our approach in intelligent robotics projects. We made the code publicly available at https://linukc.github.io/BeyondBareQueries/.