Eyes on Target: Gaze-Aware Object Detection in Egocentric Video

TL;DR

This paper introduces a gaze-aware, depth-integrated object detection framework for egocentric videos that leverages human gaze to improve detection accuracy by biasing attention mechanisms towards attended regions.

Contribution

It presents a novel method that incorporates gaze-derived features into a Vision Transformer, enhancing object detection in egocentric videos compared to traditional methods.

Findings

Gains in detection accuracy over gaze-agnostic baselines.

Effective use of gaze cues in attention mechanisms.

Improved performance on multiple egocentric datasets.

Abstract

Human gaze offers rich supervisory signals for understanding visual attention in complex visual environments. In this paper, we propose Eyes on Target, a novel depth-aware and gaze-guided object detection framework designed for egocentric videos. Our approach injects gaze-derived features into the attention mechanism of a Vision Transformer (ViT), effectively biasing spatial feature selection toward human-attended regions. Unlike traditional object detectors that treat all regions equally, our method emphasises viewer-prioritised areas to enhance object detection. We validate our method on an egocentric simulator dataset where human visual attention is critical for task assessment, illustrating its potential in evaluating human performance in simulation scenarios. We evaluate the effectiveness of our gaze-integrated model through extensive experiments and ablation studies, demonstrating consistent gains in detection accuracy over gaze-agnostic baselines on both the custom simulator dataset and public benchmarks, including Ego4D Ego-Motion and Ego-CH-Gaze datasets. To interpret model behaviour, we also introduce a gaze-aware attention head importance metric, revealing how gaze cues modulate transformer attention dynamics.