HERO-VQL: Hierarchical, Egocentric and Robust Visual Query Localization

TL;DR

HERO-VQL introduces a hierarchical, robust approach for egocentric visual query localization, leveraging attention guidance and augmentation techniques to improve accuracy amidst viewpoint changes and occlusions in egocentric videos.

Contribution

It presents a novel method combining top-down attention and augmentation-based training to enhance localization in challenging egocentric video scenarios.

Findings

Significantly outperforms baseline methods on VQ2D dataset.

Effectively handles viewpoint changes and occlusions.

Improves localization stability across diverse conditions.

Abstract

In this work, we tackle the egocentric visual query localization (VQL), where a model should localize the query object in a long-form egocentric video. Frequent and abrupt viewpoint changes in egocentric videos cause significant object appearance variations and partial occlusions, making it difficult for existing methods to achieve accurate localization. To tackle these challenges, we introduce Hierarchical, Egocentric and RObust Visual Query Localization (HERO-VQL), a novel method inspired by human cognitive process in object recognition. We propose i) Top-down Attention Guidance (TAG) and ii) Egocentric Augmentation based Consistency Training (EgoACT). Top-down Attention Guidance refines the attention mechanism by leveraging the class token for high-level context and principal component score maps for fine-grained localization. To enhance learning in diverse and challenging matching scenarios, EgoAug enhances query diversity by replacing the query with a randomly selected corresponding object from groundtruth annotations and simulates extreme viewpoint changes by reordering video frames. Additionally, CT loss enforces stable object localization across different augmentation scenarios. Extensive experiments on VQ2D dataset validate that HERO-VQL effectively handles egocentric challenges, significantly outperforming baselines.

Figures

Tables

Table 1. Table 1: Comparison with state-of-the-art on VQ2D  [ Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al. ] . We report the tAP 25 \text{tAP}_{25} , stAP 25 \text{stAP}_{25} , recovery (Rec. %), and the success rate (Succ.) on the validation set and the test set.

	Validation Set				Test Set
Method	${\text{tAP}}_{25}$	${\text{stAP}}_{25}$	Rec. %	Succ.	${\text{tAP}}_{25}$	${\text{stAP}}_{25}$	Rec. %	Succ.
SiamRCNN [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.]	0.20	0.12	32.2	39.8	0.21	0.13	34.0	41.6
NFM [Xu et al.(2022)Xu, Fu, Li, Ghanem, Perez-Rua, and Xiang]	0.26	0.19	37.9	47.9	0.24	0.17	38.6	47.7
CocoFormer [Xu et al.(2023)Xu, Li, Fu, Ghanem, Xiang, and Pérez-Rúa]	0.26	0.19	37.7	47.7	0.26	0.18	43.2	48.1
VQLoC  [Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman]	0.31	0.22	47.1	55.9	0.32	0.24	45.1	55.9
HERO-VQL (ours)	0.38	0.28	44.9	61.1	0.37	0.28	45.3	60.7

Table 2. Table 2: Ablation study. To validate the effect of each component, we show the results on the VQ2D validation set. We report tAP 25 \text{tAP}_{25} , stAP 25 \text{stAP}_{25} , recovery, and success rate as percentages.

Components		${\text{tAP}}_{25}$	${\text{stAP}}_{25}$	Rec. %	Succ.
HERO-VQL		37.5	27.6	44.9	61.1
w/o. EgoACT		34.7	26.5	45.1	59.8
w/o. TAG		33.4	25.8	43.6	59.1
Baseline		29.2	21.9	41.8	54.8

Table 3. Table 3: Hyperparameters used for training on VQ2D dataset.

Config	VQ2D [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.]
Optimizer	AdamW [Loshchilov and Hutter(2019)]
Backbone matmul precision	TensorFloat32
Learning rate	3e-3
Weight decay	0.005
Optimizer momentum [Chen et al.(2020a)Chen, Radford, Child, Wu, Jun, Luan, and Sutskever]	${\beta }_1,{\beta }_2=0.9,0.999$
Per-GPU batch size	3
Update frequency	1
Learning rate schedule	Linear
Warmup iterations	100
Training epochs	106
Flip augmentation	✓
RandomResizedCrop	✓
Brightness jitter	0.4
Contrast jitter	0.4
Saturation jitter	0.3
$\beta$	$0.1\overline{6}$
$\gamma$	$0.1\overline{6}$
$\lambda$	$0.\overline{6}$
$\mu$	0.1
QueryAug probability $p$	0.5

Table 4. Table 4: Effect of Attention Score Repair.

Method		${\text{tAP}}_{25}$	${\text{stAP}}_{25}$	Rec. %	Succ.
HERO-VQL with repair		37.5	27.6	44.9	61.1
HERO-VQL without repair		34.3	25.2	44.3	59.5

Table 5. Table 5: QueryAug hyperparameter analysis.

Probability p		${\text{tAP}}_{25}$	${\text{stAP}}_{25}$	Rec. %	Succ.
1.0		34.1	24.4	41.5	57.9
0.75		33.6	25.4	44.6	59.1
0.5		37.5	27.6	44.9	61.1
0.25		33.9	25.7	43.7	59.2
w/o. QueryAug		33.9	25.5	44.1	58.4

Equations

$${\alpha }_{\text{high}}=\sigma \left({\mathbf{Z}}_{\text{cls}}^{L-1}{\mathbf{W}}_Q({\mathbf{z}}_{\text{video}}^{L-1}{\mathbf{W}}_K{)}^{\top }\right)\in {\mathbb{R}}^M,$$

$${\alpha }_{\text{mid}}=\varphi ({\tilde{\mathbf{Z}}}_{\text{query}}{\mathbf{V}}_{{\tilde{\mathbf{Z}}}_{\text{query}}})\in [0,1{)}^{N\times R},\text{where }\varphi (x)=1-e^{-x^2/\tau }.$$

$$L=\beta L_{\text{task}}(\mathbf{C},\hat{\mathbf{C}})+\gamma L_{\text{task}}({\mathbf{C}}^{\prime },{\hat{\mathbf{C}}}^{\prime })+\lambda L_{\text{CT}}+\mu L_{\text{TAG}},$$

$$c_x=\frac{x_1+x_2}{2},c_y=\frac{y_1+y_2}{2}$$

$$w=x_2-x_1,h=y_2-y_1$$

$$\Delta x=c_{x_i}-c_{x_j},\Delta y=c_{y_i}-c_{y_j}$$

$$D=\sqrt{\Delta x^2+\Delta y^2}$$

$$\Delta w=w_i-w_j,\Delta h=h_i-h_j$$

$$V_w=(w_i>0)\wedge (w_j>0),V_h=(h_i>0)\wedge (h_j>0)$$

$$S_w=\{\begin{array}{ll}\log \left(\frac{w_i}{w_j}\right),& \text{if }{V}_w\\ 0,& \text{otherwise}\end{array}$$

$$S_h=\{\begin{array}{ll}\log \left(\frac{h_i}{h_j}\right),& \text{if }{V}_h\\ 0,& \text{otherwise}\end{array}$$

$${\Delta }_{\text{total}}=D+|\Delta w|+|\Delta h|+|S_w|+|S_h|$$

$$F^{\ast }=\underset{F^{\prime }\in {\mathbb{F}}_{\text{GT}}}{\mathrm{argmax}}\text{Displacement}(F_{\text{last}},F^{\prime })$$

$${\mathbf{A}}_{\text{self}}=\text{Softmax}\left(\frac{{\mathbf{Q}}_{SA}{\mathbf{K}}_{SA}^{\top }}{\sqrt{D}}+{\alpha }_{\text{high}}\right),$$

$${\alpha }_{\text{mid}}^{r^{\prime }}=\left[\begin{array}{c}{\alpha }_{\text{mid}}^r,{\alpha }_{\text{mid}}^r,\cdots ,{\alpha }_{\text{mid}}^r\end{array}\right]\in [0,1{)}^{M\times N}.$$

$${\mathbf{A}}_{\text{cross}}^r=\text{Softmax}\left(\frac{{\mathbf{Q}}_{CA}^r{\mathbf{K}}_{CA}^{r\top }}{\sqrt{D/N_{\text{head}}}}+{\alpha }_{\text{mid}}^{r^{\prime }}\right),$$

$${\mathbf{S}}_{\text{TAG}}=\{{\mathbf{s}}_{\text{TAG}}^i\in [0,1{)}^R{\}}_{i=1}^N=\varphi \left({\tilde{\mathbf{Z}}}_{\text{query}}{\mathbf{V}}_{\mathbf{Y}}\right).$$

$$L_{\text{token}}$$

$$L_{\text{map}}$$

$$L_{\text{TAG}}={\lambda }_{\text{token}}{L}_{\text{token}}+{\lambda }_{\text{map}}{L}_{\text{map}},$$

Full text

1 Introduction

Egocentric visual query localization (VQL) is a challenging task that aims to spatio-temporally localize the last occurrence of a visual query object within a long-form egocentric video. In Figure 1 (a), we illustrate the egocentric VQL task in the Ego4D dataset [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.]. A successful egocentric VQL method significantly enhances everyday life through applications such as intelligent AR glasses [Danielsson et al.(2020)Danielsson, Holm, and Syberfeldt] or mobile robot assistants [Graf et al.(2009)Graf, Reiser, Hägele, Mauz, and Klein].

Egocentric videos pose significant challenges due to frequent and abrupt viewpoint changes caused by the camera wearer’s movements [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al., Dunnhofer et al.(2021)Dunnhofer, Furnari, Farinella, and Micheloni, Damen et al.(2018)Damen, Doughty, Farinella, Fidler, Furnari, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray, Xu et al.(2023)Xu, Li, Fu, Ghanem, Xiang, and Pérez-Rúa]. As illustrated in Figure 1 (b), third-person videos typically maintain stable object visibility. In contrast, egocentric videos often contain objects undergoing substantial appearance changes and occlusions due to abrupt viewpoint changes. For example, as shown in Figure 1 (c), a banana can appear drastically different depending on the camera angle and sometimes a bottle is only partially visible.

Existing egocentric visual query localization works have shown great progress by using a detector and a tracking model [Tang et al.(2023)Tang, Liang, Feiszli, and Wang, Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al., Mai et al.(2023)Mai, Hamdi, Giancola, Zhao, and Ghanem, Xu et al.(2022)Xu, Fu, Li, Ghanem, Perez-Rua, and Xiang, Xu et al.(2023)Xu, Li, Fu, Ghanem, Xiang, and Pérez-Rúa] and end-to-end learning spatio-temporal relationships [Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman]. Despite the great progress, current VQL methods still struggle to accurately localize objects due to the aforementioned challenges in egocentric videos.

Unlike existing egocentric VQL methods, humans can accurately locate objects even under frequent and abrupt viewpoint changes due to ego-motion. For precise object localization, humans follow a top-down perceptual process [Ahissar and Hochstein(2004), Hochstein and Ahissar(2002)]. They first recognize objects at a high-level and then refine their perception by focusing on details.

In this work, we introduce Hierarchical, Egocentric and RObust Visual Query Localization (HERO-VQL), a new egocentric visual query localization method. First, inspired by the human perception process, we design Top-down Attention Guidance (TAG) to enhance the attention mechanism of our spatial decoder. We begin by guiding the attention process at a high level, using the class token of the object as a query during attention with video features. This allows the model to capture the global object context, helping it recognize the object even when it appears at different scales or angles. Next, we refine attention by guiding the model to focus on object details using principal component (PC) score maps derived from the query image. The PC score maps improve the model’s ability to precisely localize objects, even under varying perspectives or partial occlusions.

Second, to enable robust matching between visual query and object instances under challenging egocentric video conditions, we propose Egocentric Augmentation based Consistency Training (EgoACT). Egocentric Augmentation (EgoAug) simulates challenging real-world ego-motions. To this end, EgoAug comprises two types of augmentations: i) replacing the query image with a randomly selected corresponding object from the ground-truth (GT) annotations, allowing the model to learn from a broader range of query-GT pairs, ii) Reordering video frames to simulate more abrupt object and camera motion. With Consistency Training (CT) loss, we penalize the model when the predictions differ across augmented clips encouraging more consistent localization. Consequently, EgoACT improves the model’s robustness to appearance variations, abrupt motions, and partial occlusions.

To validate the effectiveness of HERO-VQL, we design a set of controlled experiments on the VQ2D [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.] dataset. HERO-VQL outperforms existing VQL methods.

In this work, we make the following major contributions:

• •

We propose Top-down Attention Guidance (TAG), a hierarchical attention mechanism to refine the attention of the spatial decoder.

• •

We introduce Egocentric Augmentation based Consistency Training (EgoACT) to improve robustness against appearance changes, abrupt motion, and partial occlusions.

• •

We validate the effectiveness of HERO-VQL through extensive experiments, demonstrating strong performance on egocentric video: VQ2D [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.].

2 Related Work

Egocentric visual query localization.

With the advancement of egocentric datasets [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al., Islam et al.(2024)Islam, Ho, Yang, Nagarajan, Torresani, and Bertasius, Tang et al.(2023)Tang, Liang, Feiszli, and Wang, Grauman et al.(2024)Grauman, Westbury, Torresani, Kitani, Malik, Afouras, Ashutosh, Baiyya, Bansal, Boote, Byrne, Chavis, Chen, Cheng, Chu, Crane, Dasgupta, Dong, Escobar, Forigua, Gebreselasie, Haresh, Huang, Islam, Jain, Khirodkar, Kukreja, Liang, Liu, Majumder, Mao, Martin, Mavroudi, Nagarajan, Ragusa, Ramakrishnan, Seminara, Somayazulu, Song, Su, Xue, Zhang, Zhang, Castillo, Chen, Fu, Furuta, Gonzalez, Gupta, Hu, Huang, Huang, Khoo, Kumar, Kuo, Lakhavani, Liu, Luo, Luo, Meredith, Miller, Oguntola, Pan, Peng, Pramanick, Ramazanova, Ryan, Shan, Somasundaram, Song, Southerland, Tateno, Wang, Wang, Yagi, Yan, Yang, Yu, Zha, Zhao, Zhao, Zhu, Zhuo, Arbelaez, Bertasius, Crandall, Damen, Engel, Farinella, Furnari, Ghanem, Hoffman, Jawahar, Newcombe, Park, Rehg, Sato, Savva, Shi, Shou, and Wray, Damen et al.(2018)Damen, Doughty, Farinella, Fidler, Furnari, Kazakos, Moltisanti, Munro, Perrett, Price, and Wray, Song et al.(2023)Song, Byrne, Nagarajan, Wang, Martin, and Torresani, Goletto et al.(2024)Goletto, Nagarajan, Averta, and Damen, Kurita et al.(2023)Kurita, Katsura, and Onami], there has been significant progress in egocentric visual query localization (VQL), where the query object may come from outside the video. Key works on egocentric visual query localization include using object detector and tracking-based methods [Xu et al.(2023)Xu, Li, Fu, Ghanem, Xiang, and Pérez-Rúa, Xu et al.(2022)Xu, Fu, Li, Ghanem, Perez-Rua, and Xiang, Mai et al.(2023)Mai, Hamdi, Giancola, Zhao, and Ghanem, Tang et al.(2023)Tang, Liang, Feiszli, and Wang, Huang et al.(2023)Huang, Tian, Kumar, and Xu], transformer architectures [Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman] and data augmentations [Xu et al.(2023)Xu, Li, Fu, Ghanem, Xiang, and Pérez-Rúa, Xu et al.(2022)Xu, Fu, Li, Ghanem, Perez-Rua, and Xiang]. Despite these advancements, existing methods still struggle with accurate object localization due to the challenges inherent in egocentric videos, such as frequent and abrupt viewpoint changes, significant object appearance variations, and partial occlusions. Unlike prior approaches, we tackle these challenges by leveraging learning from diverse matching scenarios and a top-down guided attention mechanism, enabling more robust and accurate localization.

Learning high-level information for object localization.

Capturing high-level information about the target object is crucial for effective object localization. Key approaches include contrastive learning [Chen et al.(2020b)Chen, Kornblith, Norouzi, and Hinton, He et al.(2020)He, Fan, Wu, Xie, and Girshick, Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Pires, Guo, Azar, et al., Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin, Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski], attention-based representation learning [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al., Cui et al.(2022)Cui, Jiang, Wang, and Wu, Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu, Chen et al.(2023a)Chen, Kang, Zhu, Wang, Peng, and Lu, Jia et al.(2023)Jia, Yuan, He, Wu, Yu, Lin, Sun, Zhang, and Hu, Song et al.(2022)Song, Yu, Chen, and Yang, Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman] and using visual reference prompt [Zhu et al.(2023)Zhu, Lai, Chen, Wang, and Lu, Yang et al.(2024)Yang, Huang, Chai, Jiang, and Hwang, Jiang et al.(2025)Jiang, Wang, Zhao, Li, and Jiang]. While these methods have shown success, they often overlook valuable global information contained in the class token in transformers [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al., Cui et al.(2022)Cui, Jiang, Wang, and Wu, Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu, Chen et al.(2023a)Chen, Kang, Zhu, Wang, Peng, and Lu, Jia et al.(2023)Jia, Yuan, He, Wu, Yu, Lin, Sun, Zhang, and Hu, Song et al.(2022)Song, Yu, Chen, and Yang]. Additionally, contrastive learning methods require large batch sizes [Chen et al.(2020b)Chen, Kornblith, Norouzi, and Hinton, He et al.(2020)He, Fan, Wu, Xie, and Girshick, Grill et al.(2020)Grill, Strub, Altché, Tallec, Richemond, Buchatskaya, Doersch, Pires, Guo, Azar, et al., Caron et al.(2020)Caron, Misra, Mairal, Goyal, Bojanowski, and Joulin, Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski] and visual reference prompts depend on additional modality inputs — both of which can be computationally impractical for VQL tasks, especially in academic settings. In contrast to prior approaches, we leverage global cues from the class token for guiding the attention process, enabling more robust object localization in challenging egocentric videos.

Using mid-level information for object localization.

Real-world visual localization tasks often face challenges such as occlusions and appearance variations, making mid-level information crucial for refining object representations and improving robustness. Existing approaches address these challenges through attention maps [Chen et al.(2023b)Chen, Ding, Cao, Shen, Zhang, Jiang, and Ji, Zhang et al.(2018)Zhang, Wei, Kang, Yang, and Huang, Li et al.(2023)Li, Jampani, Sun, and Sevilla-Lara] and part-based region matching [Cho et al.(2015)Cho, Kwak, Schmid, and Ponce, Chen et al.(2014)Chen, Mottaghi, Liu, Fidler, Urtasun, and Yuille, Fang et al.(2018)Fang, Cao, Tai, and Lu, Li et al.(2023)Li, Jampani, Sun, and Sevilla-Lara], which have demonstrated effectiveness in handling occlusions and object appearance variations. Unlike prior methods that incorporate mid-level information through detection pipelines or attention mechanisms, we leverage principal component score maps derived from query and video feature. This allows HERO-VQL to capture fine-grained details, making it particularly well-suited for the egocentric VQL task.

3 HERO-VQL

We propose HERO-VQL, a novel egocentric visual query localization (VQL) method that integrates hierarchical attention mechanisms with training strategies specifically designed to address the challenges of egocentric VQL. In this section, we first provide an overview of the proposed method in Section 3.1. Then we provide details of i) top-down attention guidance (TAG) in Section 3.2, ii) egocentric augmentation (EgoAug) in Section 3.3, and consistency training (CT) loss in Section 3.4.

3.1 Overview

The egocentric VQL task aims to spatio-temporally localize the last occurrence of an object within a long-form egocentric video, given a query image of the object. We show an overview of the proposed method in Figure 2 (a).

Formulation.

Given a RGB video $\mathbf{I}_{\text{video}}\in\mathbb{R}^{T\times C\times H\times W}$ with $T$ frames, $C$ channels, and $H\times W$ spatial dimensions, and a query image $\mathbf{I}_{\text{query}}\in\mathbb{R}^{C\times H\times W}$ as input, the goal is to predict the temporal segment where the object last appears and localize it in each frame of this segment. Each bounding box is represented as $\mathbf{B}_{i}=\{x_{i},y_{i},w_{i},h_{i}\}$, where $(x_{i},y_{i})$ denote the top-left coordinates, and $(w_{i},h_{i})$ represent the width and height of the object in the $i$-th frame.

Egocentric Augmentation based Consistency Training.

During training, we enhance the model’s robustness to frequent and abrupt viewpoint changes in egocentric videos by using Egocentric Augmentation based Consistency Training (EgoACT), as illustrated in Figure 2 (a). EgoACT consists of two components. First, Egocentric Augmentation (EgoAug) increases query diversity and simulates abrupt egocentric motion during training. EgoAug replaces the query image $\mathbf{I}_{\text{query}}$ with another instance $\mathbf{I}_{\text{QueryAug}}$ randomly selected from the ground-truth object instances in the video. Then EgoAug reorders the input video $\mathbf{I}_{\text{video}}$ to simulate abrupt viewpoint changes, resulting in $\mathbf{I}_{\text{MotionAug}}$. Second, Consistency Training (CT) loss enforces consistency between predictions across augmented clips. We provide further details on EgoAug and CT loss in Section 3.3, and Section 3.4, respectively.

Encoder.

We use a pre-trained encoder $f_{E}(\cdot)$, based on DINOv2 [Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski], to extract features from each frame of the video $\mathbf{I}_{\text{video}}$ and the query image $\mathbf{I}_{\text{query}}$. Consequently, we obtain video features $\mathbf{Z}_{\text{video}}\in\mathbb{R}^{T\times M\times D}$ and the query features $\mathbf{Z}_{\text{query}}\in\mathbb{R}^{N\times D}$, where $D$ is the embedding dimension, and $M$ and $N$ are the number of video and query patch tokens, respectively.

Spatial Decoder.

The spatial decoder $f_{D}(\cdot)$ refines object localization between the video and the query image. We adopt a standard transformer-based architecture [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] for the spatial decoder. Specifically, the video features $\mathbf{Z}_{\text{video}}$ serve as the query, while the query features $\mathbf{Z}_{\text{query}}$ serves as the key and value. To enhance localization, we introduce Top-down Attention Guidance (TAG), inspired by human visual recognition. A high-level attention guide ($\alpha_{\text{high}}$) directs the model to focus on query-relevant video regions, while a mid-level attention guide ($\alpha_{\text{mid}}$) refines localization by emphasizing distinctive object parts. These attention maps help the decoder progressively narrow down the search space and improve localization. The decoder outputs object-aware features $\mathbf{Y}\in\mathbb{R}^{T\times M\times D}$, capturing interactions between the video and the query. We provide further details on TAG in Section 3.2.

Temporal Context Modeling.

To model temporal context, we apply a temporal module $f_{T}(\cdot)$, based on TSM [Lin et al.(2019)Lin, Gan, and Han], to the object-aware features $\mathbf{Y}$ to obtain the spatio-temporal features $\mathbf{W}=f_{T}(\mathbf{Y})\in\mathbb{R}^{T\times M\times D}$. The spatio-temporal features capture inter frame-level relationships and enrich the model’s ability to understand object movements over time.

Prediction Head.

Following prior work [Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman], HERO-VQL has a box prediction head and a score prediction head. These heads take the spatio-temporal features $\mathbf{W}$ as an input and predict a bounding box $\mathbf{B}_{i}=\{x_{i},y_{i},w_{i},h_{i}\}$ and the confidence score $\pi_{i}$ of $i$-th frame.

3.2 Top-down Attention Guidance

We propose Top-down Attention Guidance (TAG) to enhance the spatial decoder’s ability to localize objects under egocentric viewpoint changes. TAG enables the spatial decoder to process the query object in a hierarchical manner: In the early attention layers, TAG provides high-level guidance for object-level localization, helping the model capture the global information of the query. In the later attention layers, TAG refines localization with mid-level guidance, focusing on fine-grained object parts for precise matching.

3.2.1 High-level Attention Guide

High-level attention guidance captures the global object context, allowing the model to focus on query-relevant regions in the video. Inspired by human perception, where objects are first recognized at a category level before utilizing fine-grained details [Ahissar and Hochstein(2004), Hochstein and Ahissar(2002)], we leverage the class token of the query image as a semantic reference to guide the decoder’s attention. As shown in Figure 2 (b), we compute attention scores between the class token $\mathbf{Z}_{\text{cls}}^{L-1}\in\mathbb{R}^{D}$ extracted from the penultimate layer of the encoder and the patch tokens $\mathbf{z}_{\text{video}}^{L-1}\in\mathbb{R}^{M\times D}$ of the video frames . Then we compute the high-level attention guide $\alpha_{\text{high}}$ as follows:

[TABLE]

where $\sigma(\cdot)$ denotes the frame-level mean score subtraction followed by the sigmoid function, $\mathbf{W}_{Q}$ and $\mathbf{W}_{K}$ are the $D\times D$ query and key projection matrices of the penultimate layer of the encoder, respectively.

The spatial decoder adds the high-level attention guide $\alpha_{\text{high}}$ to the self-attention map computed between the query and key. The resulting attention scores are then passed through a softmax to guide the focus toward video regions that are more relevant to the query object.

3.2.2 Mid-level Attention Guide

Mid-level attention guidance refines localization by directing attention toward specific object parts. Once the object is roughly localized, humans refine their perception by focusing on distinctive local features [Ahissar and Hochstein(2004), Hochstein and Ahissar(2002)]. Following this principle, we enhance localization by directing attention to specific object parts. We extract the most discriminative structures of the query object using principal component analysis (PCA) on the query features.

Principal Component.

We first center the query features $\mathbf{Z}_{\text{query}}$ to obtain $\mathbf{\tilde{Z}}_{\text{query}}$, then we perform a low-rank principal component decomposition: $\mathbf{\tilde{Z}}_{\text{query}}\approx\mathbf{U}\Sigma\mathbf{V}^{\top}_{\mathbf{\tilde{Z}}_{\text{query}}}$. We retain the top-$R$ singular vectors, denoted as $\mathbf{V}_{\mathbf{\tilde{Z}}_{\text{query}}}\in\mathbb{R}^{D\times R}$, which capture the most significant variations in the query feature.

PC score map.

As shown in Figure 2 (b), we compute the mid-level attention guide $\alpha_{\text{mid}}$ as follows:

[TABLE]

Here, $\phi(\cdot)$ is an element-wise activation function that adjusts the influence of negative singular vector directions, $\tau$ is a scaling parameter that controls the sharpness of the calibration curve. This process generates $R$ score maps $\mathbf{S}$, each highlighting different query characteristics, e.g\bmvaOneDotgrip, or base of the pan in Figure 2.

Cross-attention in the spatial decoder.

In the spatial decoder, we incorporate the mid-level attention guide $\alpha_{\text{mid}}$ into the multi-head cross-attention mechanism by assigning each of the $R$ PC score maps to one of the $R$ attention heads. Each head adds the assigned score map to the cross-attention map before applying softmax. Since each score map encodes distinct object-part information, the heads attend to different object parts, enabling part-specific attention that benefits localization under partial visibility and viewpoint changes. For more details, please refer to the supplementary material.

3.3 Egocentric Augmentation

To address egocentric VQL challenges, we introduce Egocentric Augmentation (EgoAug), which consists of two augmentations: Visual Query Augmentation (QueryAug) and Ego-motion Augmentation (MotionAug).

QueryAug.

We propose QueryAug, a simple yet effective query augmentation technique designed to leverage the diverse correspondence signals present in ground-truth annotations explicitly. As illustrated in Figure 2 (c), given a query image $\mathbf{I}_{\text{query}}$ containing an object, we gather all instances of the object in the video, denoted by $\mathbb{I}_{\text{obj}}=\{\mathbf{I}_{1},\mathbf{I}_{2},\ldots,\mathbf{I}_{k}\}$, where $k$ is the total number of target object instances. We then randomly select one instance $\mathbf{I}_{\text{QueryAug}}$ from $\mathbb{I}_{\text{obj}}$ to replace the query image with a replacement probability $p\in[0,1]$. By incorporating QueryAug, we encourage the model to learn robust object correspondence across diverse variations, such as object pose, motion blur, and partial occlusion induced by ego-motion.

MotionAug.

We propose MotionAug, a data augmentation technique designed to simulate abrupt viewpoint changes and dynamic motion changes in egocentric videos. As illustrated in Figure 2 (c), MotionAug amplifies object motion dynamics by reordering frames within the ground-truth (GT) segment of a video. This process increases the speed of bounding box movements, inducing more abrupt viewpoint shifts and challenging localization conditions. Given a RGB video $\mathbf{I}_{\text{video}}$, we construct a reordered version $\mathbf{I}_{\text{MotionAug}}$ as follows. First, we assign the initial frame of the GT temporal segment as the first frame of $\mathbf{I}_{\text{MotionAug}}$. At each step, we select the GT frame that exhibits the largest bounding box displacement relative to the bounding box in the previous frame in the $\mathbf{I}_{\text{MotionAug}}$. We repeat this process until all GT-annotated frames are reordered. To quantify displacement, we compute variations in bounding box width, height, centroid position, and their temporal derivatives. We put further details in the supplementary material. By training with MotionAug, HERO-VQL improves robustness to abrupt object and camera motion, leading to enhanced localization stability under extreme viewpoint changes.

3.4 Training

Task loss.

We define the VQL task loss $L_{\text{task}}(\mathbf{C},\hat{\mathbf{C}})$ as a function of predicted bounding boxes and the confidence values, $\hat{\mathbf{C}}$, and the ground truth bounding boxes and the occurrence labels $\mathbf{C}$. Following prior work [Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman], the task loss consists of a box regression loss and an object score loss. The box regression loss combines $L_{1}$ loss and the generalized intersection over union (GIoU) loss [Rezatofighi et al.(2019)Rezatofighi, Tsoi, Gwak, Sadeghian, Reid, and Savarese]. For the object score loss, we employ focal loss [Lin(2017)].

Consistency training loss.

To enhance temporal stability, we introduce Consistency Training (CT) loss as illustrated in Figure 2 (c). Given the predictions $\hat{\mathbf{C}}$ from the original video $\mathbf{I}_{\text{video}}$ and the predictions $\hat{\mathbf{C}}^{\prime}$ from the frame-reordered video $\mathbf{I}_{\text{MotionAug}}$ generated by MotionAug, CT loss penalizes discrepancies between them: $L_{\text{CT}}=L_{\text{task}}(\hat{\mathbf{C}},\hat{\mathbf{C}^{\prime}})$. By enforcing CT loss, the model learns to consistently predict across different motion speeds, enhancing robustness to abrupt viewpoint and object movement variations.

Total loss.

We define the total loss function for training as follows:

[TABLE]

where $\beta$, $\gamma$, $\lambda$, and $\mu$ are hyperparameters that control the relative contributions of each loss term. We define $L_{\text{TAG}}$ as an entropy-based loss function, which helps refine the attention mechanism. For further details, please refer to the supplementary material.

4 Experimental Results

In this section, we conduct experiments to address the following research questions: (1) Does HERO-VQL effectively stabilize object localization across challenging egocentric visual query localization scenarios? (Section 4.3) (2) Are the hierarchical attention and egocentric-specific training strategies effective in egocentric VQL? (Section 4.4) (3) How much each of the key components, i.e., EgoACT and TAG, contributes to performance gain? (Section 4.4) (4) What information does TAG capture to support robust localization? (Section 3) To answer these questions, we first provide dataset details in Section 4.1, the evaluation metrics in Section 4.2. We put implementation details in the supplementary material.

4.1 Datasets

We evaluate HERO-VQL on Visual Queries 2D Localization (VQ2D) [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.] dataset. VQ2D, part of the Ego4D project, comprises egocentric videos with frequent camera motion and occlusions, with an average video length of 130 seconds. This dataset provides an extensive evaluation setup for HERO-VQL, covering egocentric object localization.

4.2 Evaluation Metrics

To evaluate VQ2D, we follow the Ego4D [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.] and use the following metrics: $\text{tAP}_{25}$, $\text{stAP}_{25}$, Rec. %, and Succ. Specifically, $\text{tAP}_{25}$ and $\text{stAP}_{25}$ denote the average precision values thresholded by spatial and spatio-temporal intersection over union (IoU), respectively. The recovery (Rec. %) is frame-level recall with a spatial IoU threshold value of 0.5, while the success rate (Succ.) represents spatio-temporal precision with a low IoU threshold value of 0.05.

4.3 Comparison with the State-of-the-Art on VQ2D

In this section, we evaluate the performance of HERO-VQL and state-of-the-art methods, as shown in Table 1. HERO-VQL achieves state-of-the-art results with the highest $\text{tAP}_{25}$ of 0.38 and $\text{stAP}_{25}$ of 0.28, surpassing the second-best method, VQLoC, by 7 points and 6 points, respectively. These results demonstrate the strong performance of the proposed method in challenging egocentric visual query localization scenarios. Moreover, HERO-VQL achieves state-of-the-art performance on the test set with a significant margin, further confirming its robustness and generalization capability. By leveraging the proposed top-down attention guidance and motion-aware augmentations tailored for egocentric scenarios, HERO-VQL effectively stabilizes object localization, even under abrupt camera movements and occlusions.

4.4 Ablation Study

We conduct ablation studies to validate the effectiveness of key components and examine design choices in HERO-VQL. To ensure a fair comparison, we train and evaluate all models under identical configurations, modifying only the specific component or design choice being tested in each experiment. We report $\text{tAP}_{25}$, $\text{stAP}_{25}$, recovery, and success rate as percentages on the VQ2D validation set to provide a comprehensive performance evaluation.

Effect of TAG and EgoACT.

To evaluate the two main components, we conduct an ablation study on two key aspects of HERO-VQL: (i) top-down attention guidance (TAG), motivated by the hierarchical nature of human visual processing [Ahissar and Hochstein(2004), Hochstein and Ahissar(2002)], and (ii) egocentric augmentation based consistency training (EgoACT), which enhances robustness to motion and appearance variation. As shown in Table 2 (a), ablating EgoACT results in a notable performance drop: $\text{tAP}_{25}$ by 2.8 points and $\text{stAP}_{25}$ by 1.1 points. The results highlight the importance of motion-aware augmentations and consistency-enforcing strategies for handling appearance variations, abrupt motion, and partial occlusions. Ablating TAG results in 4.1 points drop in $\text{tAP}_{25}$ and 1.8 points drop in $\text{stAP}_{25}$, demonstrating its effectiveness in refining localization through structured top-down attention. Finally, ablating both TAG and EgoACT leads to a significant performance degradation across all metrics (3.1–8.3 points). The results confirm that the two components work synergistically to enhance visual query localization in challenging egocentric scenarios.

Effect of high-level and mid-level attention guide.

We evaluate the efficacy of high-level and mid-level attention guide of TAG in Table 2 (b). The results indicate that both guides significantly enhance performance: High-level attention guide improves $\text{tAP}_{25}$ by 1.9 points and $\text{stAP}_{25}$ by 1.5 points, demonstrating its role in capturing global object context. Mid-level attention guide contributes a 3.5 points gain in $\text{tAP}_{25}$ and a 2.2 points gain in $\text{stAP}_{25}$, highlighting its importance in fine-grained object localization. When both components are combined, they produce a synergistic effect, achieving a strong overall performance of 37.5% in $\text{tAP}_{25}$ and 27.6% in $\text{stAP}_{25}$, validating the effectiveness of top-down attention guidance.

Effect of query selection strategies in QueryAug.

To investigate different design choices for QueryAug, we conduct an ablation study and show the results in Table 2 (c). We compare three query replacement strategies and two traditional augmentations: (1) random replacement, (2) most similar instance, (3) least similar instance, (4) photometric augmentation, and (5) geometric augmentation. QueryAug outperforms traditional augmentations, indicating that leveraging diverse object instances is more effective than low-level pixel perturbations in egocentric settings. Within QueryAug, the random replacement achieves the best performance by encouraging learning from a broader range of query–GT pairs.

Effect of frame reordering strategies in MotionAug.

To investigate different design choices in MotionAug, we conduct an ablation study and show the results in Table 2 (d). We compare three frame reordering strategies: (1) maximizing object displacement, (2) random reordering, and (3) no reordering (w/o MotionAug). Among these, maximizing object displacement achieves the best performance, demonstrating its effectiveness in enhancing localization robustness under extreme viewpoint changes and abrupt motions.

Effect of consistency training loss.

As shown in Table 2 (e), incorporating CT loss improves $\text{tAP}_{25}$ by 1.5 and $\text{stAP}_{25}$ by 1.0 points, confirming its effectiveness in enhancing temporal stability. By enforcing consistency between predictions on the original and reordered video sequences, HERO-VQL becomes more robust to appearance variations, abrupt motions, and partial occlusions, improving localization stability in challenging egocentric scenarios.

Effect of backbone architectures.

We evaluate whether HERO-VQL performs consistently across different backbone architectures in Table 2 (f). We conduct experiments with DINOv2 [Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski] ViT-B/14 and CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever] ViT-L/14. HERO-VQL consistently outperforms the baseline, demonstrating its effectiveness regardless of the backbone architecture.

4.5 Qualitative Evaluation

In Figure 3, We visualize TAG’s high-level and mid-level attention guides to understand what types of information they capture. The high-level attention guide attends more strongly to patch tokens in the video features that are associated with the object. While the attention is broadly distributed across candidate regions, the patch token corresponding to the query object receives particularly high attention. Additionally, the mid-level attention guide attends to specific object parts in the query feature. The difference in visualization reflects their roles in the decoder. The high-level attention guide is added to the self-attention map with video-only inputs, whereas the mid-level attention guide is applied to the cross-attention map, where video features act as queries and the query image features serve as keys and values. These results highlight the effectiveness of TAG in enabling robust visual query localization, effectively addressing varied challenges in egocentric videos.

5 Conclusions

In this paper, we address challenges of egocentric visual query localization (VQL) task, which include appearance variations, abrupt motions, and partial occlusions. We propose HERO-VQL, a method designed to improve localization robustness through two key components: (i) TAG refines attention through hierarchical top-down guidance, enhancing object localization under varying perspectives and partial occlusions. (ii) EgoACT, a training strategy that improves robustness by simulating challenging egocentric scenarios, i.e., query replacement and motion-aware frame reordering, and enforcing consistency in localization between the original clip and an augmented clip. Extensive experiments demonstrate that HERO-VQL significantly improves robustness in egocentric scenarios and achieves state-of-the-art performance on the challenging VQ2D benchmark.

Acknowledgements

This work was supported in part by AI R&D Division, CJ Group, the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea Government (MSIT) under grant RS-2024-00353131 (20%), RS-2021-0-02068 (Artificial Intelligence Innovation Hub, 20%), and RS-2022-00155911 (Artificial Intelligence Convergence Innovation Human Resources Development (Kyung Hee University, 20%)), Additionally, it was supported by the Institute of Information & Communications Technology Planning & Evaluation(IITP)-ITRC(Information Technology Research Center) grant funded by the Korea government (MSIT)(IITP-2025-RS-2023-00259004, 20%), and the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT)(IRIS RS-2025-02216217, 20%).

Supplementary Material

In this supplementary material, we provide comprehensive dataset/baseline and method details to complement the main paper. We organize the supplementary material as follows:

Dataset details 2. 2.

Implementation details 3. 3.

Method details 4. 4.

Comprehensive quantitative results 5. 5.

Comprehensive qualitative evaluations

Appendix A Dataset Details

In this section, we provide a detailed description of Visual Queries 2D localization (VQ2D) [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.] dataset.

Details

VQ2D dataset is a large-scale egocentric video dataset for localization that contains about 6K clips. The dataset is split into train/val/test sets, with 3.6k/1.2k/1.1k clips and 13.6k/4.5k/4.4k queries, respectively. Bounding boxes are annotated at the rate of 5 frames per second.

Metrics

The evaluation of VQ2D is based on four key metrics: $\text{tAP}_{25}$, $\text{stAP}_{25}$, Rec %, and Succ. These metrics are defined as follows:

$\text{tAP}_{25}$: Temporal Average Precision (AP) calculated at a threshold of temporal Intersection over Union (tIoU). Specifically, we evaluate $\text{tAP}_{25}$ at a tIoU threshold of 0.25, indicating how well the predicted temporal boundaries align with the ground truth. Higher values suggest better temporal localization. 2. 2.

$\text{stAP}_{25}$: Spatio-temporal Average Precision (AP) calculated at a threshold of spatio-temporal Intersection over Union (stIoU). This metric measures the overlap between the predicted and ground-truth spatio-temporal volumes, capturing both spatial accuracy and temporal alignment. 3. 3.

Recovery % (Rec %): Frame-level recall, which measures the proportion of frames correctly identified within a video. For a frame to be considered correctly recovered, the spatial IoU between the prediction and ground truth must exceed 0.5. 4. 4.

Success rate (Succ.): Spatio-temporal precision, representing the proportion of spatio-temporal predictions that meet a minimum IoU threshold of 0.05. This metric emphasizes identifying whether the prediction captures even a minimal level of overlap with the ground truth.

Appendix B Implementation Details

In this section, we provide details of our experimental setup and implementation on VQ2D dataset [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.]. We conduct all the experiments with 8 RTX 3090s or RTX A5000s. We implement the proposed method in PyTorch [Paszke et al.(2017)Paszke, Gross, Chintala, Chanan, Yang, DeVito, Lin, Desmaison, Antiga, and Lerer] and PyTorch Lightning from scratch.

B.1 Training

During training, we divide each untrimmed video into non-overlapping, fixed-length clips of 32 frames at 5 fps. Each clip contains at least one instance of the target object. We resize all frames and query images to 448 $\times$ 448 pixels, setting the longer side to 448 and applying zero-padding to the shorter side to maintain the aspect ratio. We set total loss hyperparameters as: $\beta=0.1\overline{6}$, $\gamma=0.1\overline{6}$, $\lambda=0.\overline{6}$, $\mu=0.1$, and we set the QueryAug probability $p=0.5$. We train the model for 106 epochs with a batch size of 24. We use the AdamW [Loshchilov and Hutter(2019)] optimizer with a learning rate of 0.0003 and a weight decay of 0.005. We employ a warm-up scheduling for the learning rate for the initial 1,000 iterations. We employ DINOv2 [Oquab et al.(2024)Oquab, Darcet, Moutakanni, Vo, Szafraniec, Khalidov, Fernandez, HAZIZA, Massa, El-Nouby, Assran, Ballas, Galuba, Howes, Huang, Li, Misra, Rabbat, Sharma, Synnaeve, Xu, Jegou, Mairal, Labatut, Joulin, and Bojanowski] (ViT-B/14) as an encoder and TSM [Lin et al.(2019)Lin, Gan, and Han] as a temporal module.

B.2 Inference

During inference, we split the video in the same way as in training. We concatenate the predicted bounding boxes and their corresponding scores across clips to obtain a continuous sequence of predictions throughout the video. To smooth the score sequence, we apply a median filter with a kernel size of 5. From the smoothed sequence, we first identify the peak with the highest score, denoted as $\pi_{h}$. Then we threshold the score sequence with a value $0.7\pi_{h}$ to find candidate intervals where the target object appears. For the VQ2D dataset, we consider only the last temporal segment.

B.3 Box Movement Measurement in MotionAug

We measure box variation for reordering frames in MotionAug. Given a set of bounding boxes represented by their coordinates in the form: $\begin{bmatrix}y_{1},x_{1},y_{2},x_{2}\end{bmatrix}.$ For each bounding box, the following computations are performed: The center points are calculated as:

[TABLE]

The width and height are computed as:

[TABLE]

The pairwise differences in center coordinates are:

[TABLE]

The Euclidean distance between bounding box centers is:

[TABLE]

The width and height differences between bounding boxes are:

[TABLE]

To compute scale ratios, we ensure valid width and height values:

[TABLE]

The logarithmic scale ratios are:

[TABLE]

[TABLE]

Finally, the total change $\Delta_{total}$ is computed as:

[TABLE]

B.4 Hyperparameter Settings

All experiments and ablation studies use the same settings. For encoders, CLIP [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever] ViT-L/14 processes a batch size of 2 per GPU, while other encoders use a batch size of 3 per GPU. We summarize the hyperparameters in Table 3.

Appendix C Method Details

C.1 Baseline Details

We describe the baseline architecture, which excludes all our key components, including TAG and EgoACT, as illustrated in Figure 4. The architecture consists of a pre-trained encoder $f_{E}$, a spatial decoder $f_{D}$, a temporal module and a prediction head. We extract video features $\mathbf{Z}_{\text{video}}$ and query feature $\mathbf{Z}_{\text{query}}$ from encoder. The video features act as the query, while the query feature serves as the key and value, and both are processed by the spatial decoder. The outputs of the spatial decoder are passed through the temporal module. Finally, the prediction head predicts a bounding box $\mathbf{B}_{i}=\{x_{i},y_{i},w_{i},h_{i}\}$ and the confidence score $\pi_{i}$ of $i$-th frame.

C.2 Ego-motion Augmentation

To increase camera motion, MotionAug reorders frames, transforming video into a more dynamic scenario. We select the first ground-truth frame and measure box movement from the other frames. Then, we choose the frame with the maximum displacement and place it next to the first frame. We repeat this process for each frame to amplify viewpoint changes. We summarize MotionAug in Algorithm 1.

C.3 High-level Attention Guide

As shown in Figure 5, we illustrate the sources of $\mathbf{Z}_{\text{video}}^{L-1}\mathbf{W}_{K}$ and $\mathbf{Z}_{\text{cls}}^{L-1}\mathbf{W}_{Q}$ for computing the high-level attention guide $\alpha_{\text{high}}$. We extract $\mathbf{Z}_{\text{video}}^{L-1}\in\mathbb{R}^{M\times D}$ and $\mathbf{Z}_{\text{cls}}^{L-1}\in\mathbb{R}^{D}$ from the $(L-1)$th layer, referred to as the penultimate layer of the encoder, where $D$ is the token dimension. $\mathbf{Z}_{\text{video}}^{L-1}$ represents the $M$ video patch tokens, and $\mathbf{Z}_{\text{cls}}^{L-1}$ corresponds to the class token of the query image.

To compute high-level attention guide $\alpha_{\text{high}}$, we use the query projection matrix $\mathbf{W}_{Q}\in\mathbb{R}^{D\times D}$ and the key projection matrix $\mathbf{W}_{K}\in\mathbb{R}^{D\times D}$ from the self-attention mechanism of the same layer. Before applying the projection, we refine $\mathbf{Z}_{\text{video}}^{L-1}$ to repair attention scores by addressing high-norm tokens that often dominate attention. Specifically, we replace these high-norm tokens with repaired vectors, computed as the mean of their neighboring tokens.

After repairing $\mathbf{Z}_{\text{video}}^{L-1}$, we compute high-level attention guide $\alpha_{\text{high}}$ by multiplying the key vector $\mathbf{Z}_{\text{video}}^{L-1}\mathbf{W}_{K}\in\mathbb{R}^{M\times D}$ with the query vector $\mathbf{Z}_{\text{cls}}^{L-1}\mathbf{W}_{Q}\in\mathbb{R}^{D}$. We then center the values prior to applying the sigmoid function $\sigma(\cdot)$ to normalize the attention scores.

Attention score repair.

We observe that a few high-norm vectors tend to cluster in the very top-left region of the attention maps. These high-norm vectors often dominate other tokens, leading to non-informative guidance. We simply replace each high-norm token $\mathbf{z}_{\text{hn}}$ with a repaired vector $\mathbf{z}_{\text{repair}}$, whose magnitude and direction are set to the means of the neighboring tokens of $\mathbf{z}_{\text{hn}}$. By this simple fix, we prevent the attention score distribution from becoming excessively concentrated in the non-informative region.

C.4 Visualize Top-down Attention Guidance

We visualize TAG’s high-level attention guide $\alpha_{\text{high}}$ and mid-level attention guide $\alpha_{\text{mid}}$ to understand what types of information they capture. The two guides play complementary roles: the high-level guide $\alpha_{\text{high}}$ provides global semantic cues from the query’s class token, which are injected into self-attention to bias the video patches toward object-relevant regions. In contrast, the mid-level guide $\alpha_{\text{mid}}$ captures fine-grained object parts through principal component maps and is integrated into cross-attention, where each head aligns a distinct part of the query with video patches. Together, they enable a progressive refinement from global context to local details, leading to more robust localization under viewpoint changes and occlusions.

C.5 Top-down Attention Guidance in the Spatial Decoder

We describe how the spatial decoder incorporates the TAG into its self-attention and cross-attention mechanisms.

Self-Attention with high-level attention guide

We first incorporates the high-level attention guide $\alpha_{\text{high}}$ into the self-attention mechanism as a guidance. Specifically, this attention mask modifies the self-attention computation as follows:

[TABLE]

where $\mathbf{Q}_{SA}$ and $\mathbf{K}_{SA}$ are the $M\times D$ query and key projection matrices of the decoder self-attention layer. Guided by $\alpha_{\text{high}}$, the spatial decoder is able to focus on video regions more relevant to the query object.

Cross-Attention with mid-level attention guide.

Before applying the mid-level attention guide, we replicate the $r$-th column vector of $\mathbf{S}$, $\mathbf{s}_{r}\in\mathbb{R}^{N}$, $M$ (the number of video patch tokens) times:

[TABLE]

Then the spatial decoder incorporates the mid-level attention guide $\alpha_{\text{mid}}^{r}$ into the multi-head cross-attention mechanism, where the number of principal components matches the number of attention heads $N_{\text{head}}=R$, allowing each component to guide a separate attention head:

[TABLE]

where $\mathbf{Q}_{CA}^{r}$ is the $M\times D/N_{\text{head}}$ query and $\mathbf{K}_{CA}^{r}$ is the $N\times D/N_{\text{head}}$ key matrices of the $r$-th head in the cross-attention layer. Guided by $\alpha_{\text{mid}}^{r^{\prime}}$, each head focuses on different object parts, refining object localization under partial visibility issues and viewpoint changes.

C.6 Top-down Attention Guide Loss

We introduce Top-down Attention Guide (TAG) loss to ensure structured attention across object parts, encouraging the decoder to learn diverse object representations. We construct TAG score maps by projecting the centered query feature $\mathbf{\tilde{Z}}_{\text{query}}$ onto the principal components of the object-aware features $\mathbf{Y}$:

[TABLE]

To ensure distinct object representations, we define TAG loss as a combination of token-wise entropy loss $L_{\text{token}}$ and negative map-wise entropy loss $L_{\text{map}}$.

[TABLE]

Here, $\text{H}(\cdot)$ denotes the entropy function. The token-wise entropy loss $L_{\text{token}}$ minimizes the entropy of each token’s score across different maps, encouraging diverse feature activations at each spatial location. Meanwhile, the negative map-wise entropy loss $L_{\text{map}}$ maximizes the entropy among globally average-pooled maps, promoting diverse object-level representations by balancing global attention responses and preventing over-reliance on dominant components.

The final TAG Loss is defined as:

[TABLE]

where $\lambda_{\text{token}}$ and $\lambda_{\text{map}}$ are hyperparameters balancing the two loss terms. By encouraging a structured transition from global semantics to fine-grained details, TAG Loss improves object localization robustness against occlusions and viewpoint variations.

Appendix D Comprehensive Quantitative Results

In this section, we provide additional quantitative results on the VQ2D dataset [Grauman et al.(2022)Grauman, Westbury, Byrne, Chavis, Furnari, Girdhar, Hamburger, Jiang, Liu, Liu, et al.] to complement the main paper. We demonstrate (1) the effect of attention score repair in Section D.1. (2) QueryAug hyperparameter analysis in Section D.2.

All models are trained under identical configurations, varying only the specific component or design choice being tested. We report $\text{tAP}_{25}$, $\text{stAP}_{25}$, recovery (Rec. %), and success rate (Succ.) as percentages.

D.1 Effect of attention score repair

We investigate the effect of repairing attention scores in high-level attention guide in Table 4. Applying the attention score repair improves 3.2 points in $\text{tAP}_{25}$ and 2.4 points in $\text{stAP}_{25}$. By replacing high-norm tokens with the mean of their neighbors, we prevent the attention score distribution from becoming excessively concentrated in the non-informative region and resulting in performance improvement.

D.2 QueryAug hyperparameter analysis

In Table 5, we study the effect of the probability for QueryAug. We achieve the best performance when the probability is set to 0.5. This suggests that a low probability limits the model’s exposure to diverse query-GT pairs, leading to degraded performance.

Appendix E Comprehensive Qualitative Evaluation

To better understand the effectivness of HERO-VQL, we present qualitative examples in three challenging scenarios. In Figure 7, HERO-VQL successfully localizes the object even when it is partially visible, while VQLoC struggles. In Figure 8, while VQLoC [Jiang et al.(2023)Jiang, Ramakrishnan, and Grauman] fails to localize object instances, HERO-VQL accurately localizes the object despite rapid camera movement. In Figure 9, without additional fine-tuning, HERO-VQL effectively localizes the query object in an unseen video with conditions significantly different from the training data. In contrast, VQLoC produces imprecise predictions. These results highlight the effectiveness of TAG and EgoACT in enabling robust and generalizable visual query localization, successfully addressing various challenges in egocentric videos.