Towards Visual Query Localization in the 3D World

TL;DR

This paper introduces 3DVQL, the first benchmark for 3D multimodal visual query localization, along with baseline models and a novel fusion algorithm, to advance research in 3D spatial understanding.

Contribution

It presents the 3DVQL benchmark with multimodal data, manual annotations, and a new lift-and-attention fusion method, addressing the gap in 3D visual query localization research.

Findings

Existing methods vary significantly in performance across fusion modules.

The proposed LaF algorithm outperforms baseline models.

3DVQL benchmark includes 2,002 sequences with multimodal annotations.

Abstract

Visual query localization (VQL) aims to predict the spatio-temporal response of the most recent occurrence in a sequence given a query. Currently, most research focuses on visual query localization in 2D videos, while its counterpart in 3D space has received little attention. In this paper, we make the first attempt to address visual query localization in the 3D world by introducing a novel benchmark, dubbed 3DVQL. Specifically, 3DVQL contains 2,002 sequences with around 170,000 frames and 6.4K response track segments from 38 object categories. Each sequence in 3DVQL is provided with multiple modalities, including point clouds, RGB images, and depth images, to support flexible research. To ensure high-quality annotations, each sequence is manually annotated with multiple rounds of verification and refinement. To the best of our knowledge, 3DVQL is the first benchmark for 3D multimodal visual query localization. To facilitate comparison in subsequent research, we implement a series of representative 3D multimodal VQL baselines using point clouds and RGB images. The experimental results show that existing methods exhibit significant performance variations across different fusion modules. To encourage future research, we propose a lift-and-attention fusion algorithm named LaF, which significantly outperforms existing baseline models. Our benchmark and model will be publicly released at https://github.com/wuhengliangliang/3DVQL.