Saliency Prediction with External Knowledge

TL;DR

This paper introduces GraSSNet, a novel saliency prediction model that incorporates external semantic knowledge via graph structures, improving accuracy over state-of-the-art methods across multiple benchmarks.

Contribution

It proposes a new graph-based neural network that explicitly integrates external semantic knowledge into saliency prediction models.

Findings

Outperforms existing models on four benchmarks

Effectively leverages external knowledge for improved saliency prediction

Demonstrates the importance of semantic relationships in visual attention modeling

Abstract

The last decades have seen great progress in saliency prediction, with the success of deep neural networks that are able to encode high-level semantics. Yet, while humans have the innate capability in leveraging their knowledge to decide where to look (e.g. people pay more attention to familiar faces such as celebrities), saliency prediction models have only been trained with large eye-tracking datasets. This work proposes to bridge this gap by explicitly incorporating external knowledge for saliency models as humans do. We develop networks that learn to highlight regions by incorporating prior knowledge of semantic relationships, be it general or domain-specific, depending on the task of interest. At the core of the method is a new Graph Semantic Saliency Network (GraSSNet) that constructs a graph that encodes semantic relationships learned from external knowledge. A Spatial Graph Attention Network is then developed to update saliency features based on the learned graph. Experiments show that the proposed model learns to predict saliency from the external knowledge and outperforms the state-of-the-art on four saliency benchmarks.