Temporal-Spatial Feature Pyramid for Video Saliency Detection

TL;DR

This paper introduces a 3D encoder-decoder architecture that effectively combines multi-scale, spatial, and temporal features for real-time video saliency detection, significantly outperforming existing methods.

Contribution

The paper proposes a novel 3D fully convolutional encoder-decoder model that integrates multi-level features with temporal information for improved video saliency detection.

Findings

Outperforms state-of-the-art methods on multiple benchmarks

Operates in real time with high accuracy

Effectively combines scale, space, and time features

Abstract

Multi-level features are important for saliency detection. Better combination and use of multi-level features with time information can greatly improve the accuracy of the video saliency model. In order to fully combine multi-level features and make it serve the video saliency model, we propose a 3D fully convolutional encoder-decoder architecture for video saliency detection, which combines scale, space and time information for video saliency modeling. The encoder extracts multi-scale temporal-spatial features from the input continuous video frames, and then constructs temporal-spatial feature pyramid through temporal-spatial convolution and top-down feature integration. The decoder performs hierarchical decoding of temporal-spatial features from different scales, and finally produces a saliency map from the integration of multiple video frames. Our model is simple yet effective, and can run in real time. We perform abundant experiments, and the results indicate that the well-designed structure can improve the precision of video saliency detection significantly. Experimental results on three purely visual video saliency benchmarks and six audio-video saliency benchmarks demonstrate that our method outperforms the existing state-of-the-art methods.