A Data-Driven Approach for Comparing Gaze Allocation Across Conditions

TL;DR

This paper introduces a data-driven method using deep neural networks to analyze how gaze allocation changes with sound, revealing new insights into visual attention strategies.

Contribution

The novel approach uses DNNs and reverse correlation to objectively compare gaze patterns across conditions, uncovering non-trivial fixation strategies.

Findings

DNNs classified gaze conditions with accuracy significantly higher than chance for some scenes.

Sound influences gaze allocation, with fixations sometimes avoiding sound sources and salient features.

Traditional ROI and heatmap methods failed to detect significant differences after correction.

Abstract

Gaze analysis often relies on hypothesised, subjectively defined regions of interest (ROIs) or heatmaps: ROIs enable condition comparisons but reduce objectivity and exploration; while heatmaps avoid this, they require many pixel-wise comparisons, making differences hard to detect. Here, we propose an advanced data-driven approach for analysing gaze behaviour. We use DNNs (adapted versions of AlexNet) to classify conditions from gaze patterns, paired with reverse correlation to show where and how gaze differs between conditions. We test our approach on data from an experiment investigating the effects of object-specific sounds (e.g., church bell ringing) on gaze allocation. ROI-based analysis shows a significant difference between conditions (congruent sound, no sound, phase-scrambled sound and pink noise), with more gaze allocation on sound-associated objects in the congruent sound condition. However, as expected, significance depends on the definition of the ROIs. Heatmaps show some unclear qualitative differences, but none are significant after correcting for pixelwise comparisons. We showed that, for some scenes, the DNNs could classify the task based on individual fixations with accuracy significantly higher than chance. Our approach shows that sound can alter gaze allocation, revealing task-specific, non-trivial strategies: fixations are not always drawn to the sound source but shift away from salient features, sometimes falling between salient features and the sound source. Crucially, such fixation strategies could not be revealed using a traditional hypothesis-driven approach. Overall, the method is objective, data-driven, and enables clear comparisons of conditions.

Figures