Integrated Gradient Correlation: a Dataset-wise Attribution Method

TL;DR

This paper introduces Integrated Gradient Correlation (IGC), a dataset-wide attribution method that summarizes model attribution patterns across datasets to enhance interpretability at the task level, especially when input importance localization is stable.

Contribution

The paper proposes IGC, a novel dataset-wise attribution technique that directly sums component contributions and relates them to model predictions, enabling region-specific analysis.

Findings

IGC reveals stable, region-specific attribution patterns in synthetic data.

Application of IGC to fMRI data uncovers meaningful neural activation patterns.

IGC correlates attribution sums with model prediction scores.

Abstract

Attribution methods are primarily designed to study input component contributions to individual model predictions. However, some research applications require a summary of attribution patterns across the entire dataset to facilitate the interpretability of the scrutinized models at a task-level rather than an instance-level. It specifically applies when the localization of important input information is supposed to be stable for a specific problem but remains unidentified among numerous components. In this paper, we present a dataset-wise attribution method called Integrated Gradient Correlation (IGC) that enables region-specific analysis by a direct summation over associated components, and further relates the sum of all attributions to a model prediction score (correlation). We demonstrate IGC on synthetic data and fMRI neural signals (NSD dataset) with the study of the representation of image features in the brain and the estimation of the visual receptive field of neural populations. The resulting IGC attributions reveal selective patterns, coherent with respective model objectives.