Single-Stage Visual Query Localization in Egocentric Videos

TL;DR

VQLoC introduces a fast, end-to-end single-stage framework for visual query localization in egocentric videos, significantly improving accuracy and inference speed over prior multi-stage methods.

Contribution

The paper presents VQLoC, a novel single-stage, end-to-end trainable framework that jointly models query-video relationships for efficient spatio-temporal localization.

Findings

Outperforms prior methods by 20% accuracy

Achieves 10x faster inference speed

Top entry on Ego4D VQ2D challenge leaderboard

Abstract

Visual Query Localization on long-form egocentric videos requires spatio-temporal search and localization of visually specified objects and is vital to build episodic memory systems. Prior work develops complex multi-stage pipelines that leverage well-established object detection and tracking methods to perform VQL. However, each stage is independently trained and the complexity of the pipeline results in slow inference speeds. We propose VQLoC, a novel single-stage VQL framework that is end-to-end trainable. Our key idea is to first build a holistic understanding of the query-video relationship and then perform spatio-temporal localization in a single shot manner. Specifically, we establish the query-video relationship by jointly considering query-to-frame correspondences between the query and each video frame and frame-to-frame correspondences between nearby video frames. Our experiments demonstrate that our approach outperforms prior VQL methods by 20% accuracy while obtaining a 10x improvement in inference speed. VQLoC is also the top entry on the Ego4D VQ2D challenge leaderboard. Project page: https://hwjiang1510.github.io/VQLoC/