Robust Egocentric Visual Attention Prediction Through Language-guided Scene Context-aware Learning

TL;DR

This paper introduces a language-guided, scene context-aware learning framework for egocentric visual attention prediction, improving robustness and accuracy by leveraging scene descriptions and focusing on relevant regions.

Contribution

It proposes a novel context perceiver guided by language descriptions and dual training objectives to enhance attention prediction in egocentric videos.

Findings

Achieves state-of-the-art performance on Ego4D and AEA datasets.

Demonstrates improved robustness across diverse egocentric scenarios.

Effectively suppresses irrelevant distractions in attention prediction.

Abstract

As the demand for analyzing egocentric videos grows, egocentric visual attention prediction, anticipating where a camera wearer will attend, has garnered increasing attention. However, it remains challenging due to the inherent complexity and ambiguity of dynamic egocentric scenes. Motivated by evidence that scene contextual information plays a crucial role in modulating human attention, in this paper, we present a language-guided scene context-aware learning framework for robust egocentric visual attention prediction. We first design a context perceiver which is guided to summarize the egocentric video based on a language-based scene description, generating context-aware video representations. We then introduce two training objectives that: 1) encourage the framework to focus on the target point-of-interest regions and 2) suppress distractions from irrelevant regions which are less likely to attract first-person attention. Extensive experiments on Ego4D and Aria Everyday Activities (AEA) datasets demonstrate the effectiveness of our approach, achieving state-of-the-art performance and enhanced robustness across diverse, dynamic egocentric scenarios.