# HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localization

## TL;DR

HERO-VQL introduces a hierarchical, robust approach for egocentric visual query localization, leveraging attention guidance and augmentation techniques to improve accuracy amidst viewpoint changes and occlusions in egocentric videos.

## Contribution

It presents a novel method combining top-down attention and augmentation-based training to enhance localization in challenging egocentric video scenarios.

## Key findings

- Significantly outperforms baseline methods on VQ2D dataset.
- Effectively handles viewpoint changes and occlusions.
- Improves localization stability across diverse conditions.

## Abstract

In this work, we tackle the egocentric visual query localization (VQL), where a model should localize the query object in a long-form egocentric video. Frequent and abrupt viewpoint changes in egocentric videos cause significant object appearance variations and partial occlusions, making it difficult for existing methods to achieve accurate localization. To tackle these challenges, we introduce Hierarchical, Egocentric and RObust Visual Query Localization (HERO-VQL), a novel method inspired by human cognitive process in object recognition. We propose i) Top-down Attention Guidance (TAG) and ii) Egocentric Augmentation based Consistency Training (EgoACT). Top-down Attention Guidance refines the attention mechanism by leveraging the class token for high-level context and principal component score maps for fine-grained localization. To enhance learning in diverse and challenging matching scenarios, EgoAug enhances query diversity by replacing the query with a randomly selected corresponding object from groundtruth annotations and simulates extreme viewpoint changes by reordering video frames. Additionally, CT loss enforces stable object localization across different augmentation scenarios. Extensive experiments on VQ2D dataset validate that HERO-VQL effectively handles egocentric challenges, significantly outperforming baselines.