SESS: Saliency Enhancing with Scaling and Sliding

TL;DR

SESS is a versatile method that enhances saliency maps by combining multi-scale, multi-region saliency information, making them more robust and discriminative for applications like object detection and explainable AI.

Contribution

The paper introduces SESS, a novel, model-agnostic approach that improves saliency maps through multi-scale fusion and an efficient pre-filtering step, addressing scale variance and distractors.

Findings

Significant improvement in saliency quality on object recognition benchmarks

Enhanced robustness of saliency maps to scale and distractors

Efficient pre-filtering reduces computational load while maintaining performance

Abstract

High-quality saliency maps are essential in several machine learning application areas including explainable AI and weakly supervised object detection and segmentation. Many techniques have been developed to generate better saliency using neural networks. However, they are often limited to specific saliency visualisation methods or saliency issues. We propose a novel saliency enhancing approach called SESS (Saliency Enhancing with Scaling and Sliding). It is a method and model agnostic extension to existing saliency map generation methods. With SESS, existing saliency approaches become robust to scale variance, multiple occurrences of target objects, presence of distractors and generate less noisy and more discriminative saliency maps. SESS improves saliency by fusing saliency maps extracted from multiple patches at different scales from different areas, and combines these individual maps using a novel fusion scheme that incorporates channel-wise weights and spatial weighted average. To improve efficiency, we introduce a pre-filtering step that can exclude uninformative saliency maps to improve efficiency while still enhancing overall results. We evaluate SESS on object recognition and detection benchmarks where it achieves significant improvement. The code is released publicly to enable researchers to verify performance and further development. Code is available at: https://github.com/neouyghur/SESS