Simple vs complex temporal recurrences for video saliency prediction

TL;DR

This study compares simple and complex recurrence methods in neural networks for video saliency prediction, demonstrating that both approaches achieve state-of-the-art results using pre-trained models and fine-tuning.

Contribution

It introduces and evaluates two recurrence modifications—ConvLSTM and exponential moving average—for temporal integration in saliency prediction models.

Findings

Both methods achieve state-of-the-art results.

The simpler exponential moving average performs comparably to ConvLSTM.

Pre-trained models fine-tuned on DHF1K improve performance.

Abstract

This paper investigates modifying an existing neural network architecture for static saliency prediction using two types of recurrences that integrate information from the temporal domain. The first modification is the addition of a ConvLSTM within the architecture, while the second is a conceptually simple exponential moving average of an internal convolutional state. We use weights pre-trained on the SALICON dataset and fine-tune our model on DHF1K. Our results show that both modifications achieve state-of-the-art results and produce similar saliency maps. Source code is available at https://git.io/fjPiB.