PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localization

TL;DR

PRVQL introduces a progressive, knowledge-guided refinement framework for egocentric visual query localization, effectively handling appearance changes and clutter by iteratively improving target features and localization accuracy.

Contribution

It proposes a novel multi-stage framework that exploits target-relevant knowledge from videos to progressively refine features for robust localization in egocentric videos.

Findings

Achieves state-of-the-art results on Ego4D dataset.

Significantly outperforms previous methods in complex scenes.

Demonstrates effective knowledge-guided feature refinement.

Abstract

Egocentric visual query localization (EgoVQL) focuses on localizing the target of interest in space and time from first-person videos, given a visual query. Despite recent progressive, existing methods often struggle to handle severe object appearance changes and cluttering background in the video due to lacking sufficient target cues, leading to degradation. Addressing this, we introduce PRVQL, a novel Progressive knowledge-guided Refinement framework for EgoVQL. The core is to continuously exploit target-relevant knowledge directly from videos and utilize it as guidance to refine both query and video features for improving target localization. Our PRVQL contains multiple processing stages. The target knowledge from one stage, comprising appearance and spatial knowledge extracted via two specially designed knowledge learning modules, are utilized as guidance to refine the query and videos features for the next stage, which are used to generate more accurate knowledge for further feature refinement. With such a progressive process, target knowledge in PRVQL can be gradually improved, which, in turn, leads to better refined query and video features for localization in the final stage. Compared to previous methods, our PRVQL, besides the given object cues, enjoys additional crucial target information from a video as guidance to refine features, and hence enhances EgoVQL in complicated scenes. In our experiments on challenging Ego4D, PRVQL achieves state-of-the-art result and largely surpasses other methods, showing its efficacy. Our code, model and results will be released at https://github.com/fb-reps/PRVQL.