Invariance Analysis of Saliency Models versus Human Gaze During Scene Free Viewing

TL;DR

This study examines how various distortions affect human gaze and saliency models, revealing significant performance drops and proposing data augmentation strategies to improve model robustness.

Contribution

It introduces a large-scale database of eye movements on distorted images and systematically analyzes the impact of distortions on saliency prediction and gaze behavior.

Findings

Saliency model performance drops significantly on distorted images.

Rotation and shearing distortions cause the largest performance decline.

Certain data augmentation methods can enhance model robustness against distortions.

Abstract

Most of current studies on human gaze and saliency modeling have used high-quality stimuli. In real world, however, captured images undergo various types of distortions during the whole acquisition, transmission, and displaying chain. Some distortion types include motion blur, lighting variations and rotation. Despite few efforts, influences of ubiquitous distortions on visual attention and saliency models have not been systematically investigated. In this paper, we first create a large-scale database including eye movements of 10 observers over 1900 images degraded by 19 types of distortions. Second, by analyzing eye movements and saliency models, we find that: a) observers look at different locations over distorted versus original images, and b) performances of saliency models are drastically hindered over distorted images, with the maximum performance drop belonging to Rotation and Shearing distortions. Finally, we investigate the effectiveness of different distortions when serving as data augmentation transformations. Experimental results verify that some useful data augmentation transformations which preserve human gaze of reference images can improve deep saliency models against distortions, while some invalid transformations which severely change human gaze will degrade the performance.