Deep saliency: What is learnt by a deep network about saliency?

TL;DR

This paper investigates what deep neural networks learn about saliency by visualizing neuron receptive fields, revealing how fine-tuning transforms representations to resemble classical saliency filters.

Contribution

It demonstrates how fine-tuning a pre-trained CNN on saliency detection induces transformations in deeper layers, producing receptive fields similar to classical center-surround filters.

Findings

Fine-tuning alters deep layer representations significantly.

Receptive fields evolve to resemble classical saliency filters.

Deep networks learn interpretable features related to saliency.

Abstract

Deep convolutional neural networks have achieved impressive performance on a broad range of problems, beating prior art on established benchmarks, but it often remains unclear what are the representations learnt by those systems and how they achieve such performance. This article examines the specific problem of saliency detection, where benchmarks are currently dominated by CNN-based approaches, and investigates the properties of the learnt representation by visualizing the artificial neurons' receptive fields. We demonstrate that fine tuning a pre-trained network on the saliency detection task lead to a profound transformation of the network's deeper layers. Moreover we argue that this transformation leads to the emergence of receptive fields conceptually similar to the centre-surround filters hypothesized by early research on visual saliency.