Deep Learning for Saliency Prediction in Natural Video

TL;DR

This paper introduces a deep learning approach for saliency prediction in natural videos, leveraging residual motion sensitivity and contrast features, achieving high accuracy and computational efficiency improvements.

Contribution

It extends deep saliency prediction methods from still images to videos by incorporating motion sensitivity and contrast features, with a novel data selection strategy for faster computation.

Findings

Achieved 89.51% accuracy on IRCCYN dataset.

Improved saliency prediction accuracy by up to 2% over RGB-only methods.

Increased AUC metric by up to 16% on video clips.

Abstract

The purpose of this paper is the detection of salient areas in natural video by using the new deep learning techniques. Salient patches in video frames are predicted first. Then the predicted visual fixation maps are built upon them. We design the deep architecture on the basis of CaffeNet implemented with Caffe toolkit. We show that changing the way of data selection for optimisation of network parameters, we can save computation cost up to 12 times. We extend deep learning approaches for saliency prediction in still images with RGB values to specificity of video using the sensitivity of the human visual system to residual motion. Furthermore, we complete primary colour pixel values by contrast features proposed in classical visual attention prediction models. The experiments are conducted on two publicly available datasets. The first is IRCCYN video database containing 31 videos with an overall amount of 7300 frames and eye fixations of 37 subjects. The second one is HOLLYWOOD2 provided 2517 movie clips with the eye fixations of 19 subjects. On IRCYYN dataset, the accuracy obtained is of 89.51%. On HOLLYWOOD2 dataset, results in prediction of saliency of patches show the improvement up to 2% with regard to RGB use only. The resulting accuracy of 76, 6% is obtained. The AUC metric in comparison of predicted saliency maps with visual fixation maps shows the increase up to 16% on a sample of video clips from this dataset.