The Roles of Contextual Semantic Relevance Metrics in Human Visual Processing

TL;DR

This paper introduces and evaluates contextual semantic relevance metrics derived from deep learning models, demonstrating their effectiveness in predicting human visual fixation and processing by integrating visual and language-based information.

Contribution

It proposes a novel combined semantic relevance metric that integrates vision-based and language-based data, advancing understanding of their roles in human visual perception.

Findings

All metrics accurately predict fixation measures.

The combined metric outperforms individual metrics.

Semantic relevance metrics simulate top-down and bottom-up processing.

Abstract

Semantic relevance metrics can capture both the inherent semantics of individual objects and their relationships to other elements within a visual scene. Numerous previous research has demonstrated that these metrics can influence human visual processing. However, these studies often did not fully account for contextual information or employ the recent deep learning models for more accurate computation. This study investigates human visual perception and processing by introducing the metrics of contextual semantic relevance. We evaluate semantic relationships between target objects and their surroundings from both vision-based and language-based perspectives. Testing a large eye-movement dataset from visual comprehension, we employ state-of-the-art deep learning techniques to compute these metrics and analyze their impacts on fixation measures on human visual processing through advanced statistical models. These metrics could also simulate top-down and bottom-up processing in visual perception. This study further integrates vision-based and language-based metrics into a novel combined metric, addressing a critical gap in previous research that often treated visual and semantic similarities separately. Results indicate that all metrics could precisely predict fixation measures in visual perception and processing, but with distinct roles in prediction. The combined metric outperforms other metrics, supporting theories that emphasize the interaction between semantic and visual information in shaping visual perception/processing. This finding aligns with growing recognition of the importance of multi-modal information processing in human cognition. These insights enhance our understanding of cognitive mechanisms underlying visual processing and have implications for developing more accurate computational models in fields such as cognitive science and human-computer interaction.