AV-Gaze: A Study on the Effectiveness of Audio Guided Visual Attention Estimation for Non-Profilic Faces

TL;DR

This paper investigates how audio signals can improve visual attention and gaze estimation in challenging conditions, using a cross-modal weak-supervision approach that leverages audio, visual, or both modalities for inference.

Contribution

It introduces AV-Gaze, a novel framework that combines audio and visual data to enhance gaze estimation under difficult real-world scenarios, with adaptive modality usage.

Findings

Achieves competitive results on multiple benchmark datasets.

Enhances gaze estimation accuracy in challenging conditions.

Demonstrates high adaptability to different modalities and scenarios.

Abstract

In challenging real-life conditions such as extreme head-pose, occlusions, and low-resolution images where the visual information fails to estimate visual attention/gaze direction, audio signals could provide important and complementary information. In this paper, we explore if audio-guided coarse head-pose can further enhance visual attention estimation performance for non-prolific faces. Since it is difficult to annotate audio signals for estimating the head-pose of the speaker, we use off-the-shelf state-of-the-art models to facilitate cross-modal weak-supervision. During the training phase, the framework learns complementary information from synchronized audio-visual modality. Our model can utilize any of the available modalities i.e. audio, visual or audio-visual for task-specific inference. It is interesting to note that, when AV-Gaze is tested on benchmark datasets with these specific modalities, it achieves competitive results on multiple datasets, while being highly adaptive toward challenging scenarios.