Do Saliency Models Detect Odd-One-Out Targets? New Datasets and Evaluations

TL;DR

This paper evaluates whether current saliency models can detect odd-one-out targets, introducing new datasets and showing that most models fail to identify singleton targets effectively, even after training.

Contribution

It introduces two novel datasets for testing singleton detection and demonstrates that existing saliency models, including CNN-based ones, do not effectively detect odd-one-out targets.

Findings

Most saliency algorithms fail to detect singleton targets.

Training CNN models on these datasets does not significantly improve detection.

New datasets enable better evaluation of saliency models' ability to detect salient singleton targets.

Abstract

Recent advances in the field of saliency have concentrated on fixation prediction, with benchmarks reaching saturation. However, there is an extensive body of works in psychology and neuroscience that describe aspects of human visual attention that might not be adequately captured by current approaches. Here, we investigate singleton detection, which can be thought of as a canonical example of salience. We introduce two novel datasets, one with psychophysical patterns and one with natural odd-one-out stimuli. Using these datasets we demonstrate through extensive experimentation that nearly all saliency algorithms do not adequately respond to singleton targets in synthetic and natural images. Furthermore, we investigate the effect of training state-of-the-art CNN-based saliency models on these types of stimuli and conclude that the additional training data does not lead to a significant improvement of their ability to find odd-one-out targets. Datasets are available at http://data.nvision2.eecs.yorku.ca/P3O3/.