Personalized Saliency and its Prediction

TL;DR

This paper introduces a method to predict personalized visual saliency maps by decomposing universal saliency into a common component and a personalized discrepancy, using novel neural network models trained on a new dataset.

Contribution

It proposes a new approach to model personalized saliency by decomposing it into universal and individual-specific components, along with two neural network solutions and a new dataset.

Findings

Models effectively predict personalized saliency maps.

The approach generalizes well to unseen observers.

The proposed methods outperform baseline models.

Abstract

Nearly all existing visual saliency models by far have focused on predicting a universal saliency map across all observers. Yet psychology studies suggest that visual attention of different observers can vary significantly under specific circumstances, especially a scene is composed of multiple salient objects. To study such heterogenous visual attention pattern across observers, we first construct a personalized saliency dataset and explore correlations between visual attention, personal preferences, and image contents. Specifically, we propose to decompose a personalized saliency map (referred to as PSM) into a universal saliency map (referred to as USM) predictable by existing saliency detection models and a new discrepancy map across users that characterizes personalized saliency. We then present two solutions towards predicting such discrepancy maps, i.e., a multi-task convolutional neural network (CNN) framework and an extended CNN with Person-specific Information Encoded Filters (CNN-PIEF). Extensive experimental results demonstrate the effectiveness of our models for PSM prediction as well their generalization capability for unseen observers.