SG-FCN: A Motion and Memory-Based Deep Learning Model for Video Saliency Detection

TL;DR

This paper introduces SG-FCN, a deep learning model that integrates motion and memory mechanisms to enhance video saliency detection, outperforming existing methods across multiple datasets.

Contribution

The paper presents a novel fully convolutional network that simulates human memory and attention to improve video fixation detection performance.

Findings

Outperforms 11 state-of-the-art methods on multiple datasets

Effectively integrates motion and temporal information

Demonstrates superior accuracy in complex scenes

Abstract

Data-driven saliency detection has attracted strong interest as a result of applying convolutional neural networks to the detection of eye fixations. Although a number of imagebased salient object and fixation detection models have been proposed, video fixation detection still requires more exploration. Different from image analysis, motion and temporal information is a crucial factor affecting human attention when viewing video sequences. Although existing models based on local contrast and low-level features have been extensively researched, they failed to simultaneously consider interframe motion and temporal information across neighboring video frames, leading to unsatisfactory performance when handling complex scenes. To this end, we propose a novel and efficient video eye fixation detection model to improve the saliency detection performance. By simulating the memory mechanism and visual attention mechanism of human beings when watching a video, we propose a step-gained fully convolutional network by combining the memory information on the time axis with the motion information on the space axis while storing the saliency information of the current frame. The model is obtained through hierarchical training, which ensures the accuracy of the detection. Extensive experiments in comparison with 11 state-of-the-art methods are carried out, and the results show that our proposed model outperforms all 11 methods across a number of publicly available datasets.