Toward Faithful Segmentation Attribution via Benchmarking and Dual-Evidence Fusion

TL;DR

This paper introduces a new benchmark for evaluating the faithfulness of segmentation attribution methods and proposes Dual-Evidence Attribution (DEA), which fuses gradient and intervention signals to improve attribution faithfulness and robustness.

Contribution

The paper presents a reproducible benchmark for segmentation attribution evaluation and introduces DEA, a novel fusion method that enhances attribution faithfulness and robustness.

Findings

DEA improves deletion-based faithfulness over gradient-only methods.

The benchmark reveals a faithfulness-stability tradeoff among attribution methods.

DEA maintains robustness with additional computational cost.

Abstract

Attribution maps for semantic segmentation are almost always judged by visual plausibility. Yet looking convincing does not guarantee that the highlighted pixels actually drive the model's prediction, nor that attribution credit stays within the target region. These questions require a dedicated evaluation protocol. We introduce a reproducible benchmark that tests intervention-based faithfulness, off-target leakage, perturbation robustness, and runtime on Pascal VOC and SBD across three pretrained backbones. To further demonstrate the benchmark, we propose Dual-Evidence Attribution (DEA), a lightweight correction that fuses gradient evidence with region-level intervention signals through agreement-weighted fusion. DEA increases emphasis where both sources agree and retains causal support when gradient responses are unstable. Across all completed runs, DEA consistently improves deletion-based faithfulness over gradient-only baselines and preserves strong robustness, at the cost of additional compute from intervention passes. The benchmark exposes a faithfulness-stability tradeoff among attribution families that is entirely hidden under visual evaluation, providing a foundation for principled method selection in segmentation explainability. Code is available at https://github.com/anmspro/DEA.