ObjectVisA-120: Object-based Visual Attention Prediction in Interactive Street-crossing Environments

TL;DR

This paper introduces ObjectVisA-120, a comprehensive VR dataset for object-based visual attention in street-crossing scenarios, along with a new evaluation metric and a novel graph-based attention prediction model, advancing understanding and modeling of human attention.

Contribution

The paper provides a new VR dataset with rich annotations for object-based attention, a novel metric oSIM for evaluation, and a graph-encoded attention model, SUMGraph, improving attention prediction performance.

Findings

Optimizing for object-based attention improves oSIM scores.

SUMGraph outperforms state-of-the-art models in attention prediction.

ObjectVisA-120 enables better evaluation of object-based attention models.

Abstract

The object-based nature of human visual attention is well-known in cognitive science, but has only played a minor role in computational visual attention models so far. This is mainly due to a lack of suitable datasets and evaluation metrics for object-based attention. To address these limitations, we present ObjectVisA-120 -- a novel 120-participant dataset of spatial street-crossing navigation in virtual reality specifically geared to object-based attention evaluations. The uniqueness of the presented dataset lies in the ethical and safety affiliated challenges that make collecting comparable data in real-world environments highly difficult. ObjectVisA-120 not only features accurate gaze data and a complete state-space representation of objects in the virtual environment, but it also offers variable scenario complexities and rich annotations, including panoptic segmentation, depth information, and vehicle keypoints. We further propose object-based similarity (oSIM) as a novel metric to evaluate the performance of object-based visual attention models, a previously unexplored performance characteristic. Our evaluations show that explicitly optimising for object-based attention not only improves oSIM performance but also leads to an improved model performance on common metrics. In addition, we present SUMGraph, a Mamba U-Net-based model, which explicitly encodes critical scene objects (vehicles) in a graph representation, leading to further performance improvements over several state-of-the-art visual attention prediction methods. The dataset, code and models will be publicly released.