Geometrically Guided Integrated Gradients

TL;DR

This paper introduces a novel interpretability method called geometrically-guided integrated gradients that leverages local geometric properties of the model parameter space to improve explanations of deep neural network predictions.

Contribution

It proposes a new explanation technique that explores model behavior across scaled inputs, outperforming existing gradient-based methods in accuracy and robustness.

Findings

Outperforms vanilla and integrated gradients in subjective assessments.

Provides a new sanity check called 'model perturbation'.

Demonstrates improved attribution quality through extensive experiments.

Abstract

Interpretability methods for deep neural networks mainly focus on the sensitivity of the class score with respect to the original or perturbed input, usually measured using actual or modified gradients. Some methods also use a model-agnostic approach to understanding the rationale behind every prediction. In this paper, we argue and demonstrate that local geometry of the model parameter space relative to the input can also be beneficial for improved post-hoc explanations. To achieve this goal, we introduce an interpretability method called "geometrically-guided integrated gradients" that builds on top of the gradient calculation along a linear path as traditionally used in integrated gradient methods. However, instead of integrating gradient information, our method explores the model's dynamic behavior from multiple scaled versions of the input and captures the best possible attribution for each input. We demonstrate through extensive experiments that the proposed approach outperforms vanilla and integrated gradients in subjective and quantitative assessment. We also propose a "model perturbation" sanity check to complement the traditionally used "model randomization" test.