Improving saliency models' predictions of the next fixation with humans' intrinsic cost of gaze shifts

TL;DR

This paper introduces a new framework that enhances saliency models by incorporating human gaze costs and sequential decision-making, significantly improving predictions of where humans will look next.

Contribution

It presents a novel algorithm that converts static saliency maps into dynamic, history-dependent predictions using human gaze costs and exploration bonuses, outperforming existing models.

Findings

Significant improvement in NSS and AUC scores across datasets

Effective integration of human gaze cost functions into saliency prediction

Outperforms five state-of-the-art saliency models on multiple datasets

Abstract

The human prioritization of image regions can be modeled in a time invariant fashion with saliency maps or sequentially with scanpath models. However, while both types of models have steadily improved on several benchmarks and datasets, there is still a considerable gap in predicting human gaze. Here, we leverage two recent developments to reduce this gap: theoretical analyses establishing a principled framework for predicting the next gaze target and the empirical measurement of the human cost for gaze switches independently of image content. We introduce an algorithm in the framework of sequential decision making, which converts any static saliency map into a sequence of dynamic history-dependent value maps, which are recomputed after each gaze shift. These maps are based on 1) a saliency map provided by an arbitrary saliency model, 2) the recently measured human cost function quantifying preferences in magnitude and direction of eye movements, and 3) a sequential exploration bonus, which changes with each subsequent gaze shift. The parameters of the spatial extent and temporal decay of this exploration bonus are estimated from human gaze data. The relative contributions of these three components were optimized on the MIT1003 dataset for the NSS score and are sufficient to significantly outperform predictions of the next gaze target on NSS and AUC scores for five state of the art saliency models on three image data sets. Thus, we provide an implementation of human gaze preferences, which can be used to improve arbitrary saliency models' predictions of humans' next gaze targets.