AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring

TL;DR

AugRefer enhances 3D visual grounding by generating diverse training data through cross-modal augmentation and improving object referencing with a language-spatial adaptive decoder, leading to better performance on benchmarks.

Contribution

It introduces a novel cross-modal augmentation method and a language-spatial adaptive decoder to improve 3D visual grounding performance.

Findings

Significant performance improvements on three benchmark datasets.

Effective generation of diverse text-3D pairs for training.

Enhanced utilization of spatial relations in 3D grounding.

Abstract

3D visual grounding (3DVG), which aims to correlate a natural language description with the target object within a 3D scene, is a significant yet challenging task. Despite recent advancements in this domain, existing approaches commonly encounter a shortage: a limited amount and diversity of text3D pairs available for training. Moreover, they fall short in effectively leveraging different contextual clues (e.g., rich spatial relations within the 3D visual space) for grounding. To address these limitations, we propose AugRefer, a novel approach for advancing 3D visual grounding. AugRefer introduces cross-modal augmentation designed to extensively generate diverse text-3D pairs by placing objects into 3D scenes and creating accurate and semantically rich descriptions using foundation models. Notably, the resulting pairs can be utilized by any existing 3DVG methods for enriching their training data. Additionally, AugRefer presents a language-spatial adaptive decoder that effectively adapts the potential referring objects based on the language description and various 3D spatial relations. Extensive experiments on three benchmark datasets clearly validate the effectiveness of AugRefer.