OrdShap: Feature Position Importance for Sequential Black-Box Models

TL;DR

OrdShap is a new feature attribution method for sequential models that separates the effects of feature values and their positions, providing more nuanced insights into model predictions.

Contribution

It introduces a theoretically grounded approach to position-sensitive attribution by disentangling feature value and position effects in sequential models.

Findings

Effective in health, natural language, and synthetic datasets.

Captures both feature value and position attributions.

Provides deeper insights into model behavior.

Abstract

Sequential deep learning models excel in domains with temporal or sequential dependencies, but their complexity necessitates post-hoc feature attribution methods for understanding their predictions. While existing techniques quantify feature importance, they inherently assume fixed feature ordering - conflating the effects of (1) feature values and (2) their positions within input sequences. To address this gap, we introduce OrdShap, a novel attribution method that disentangles these effects by quantifying how a model's predictions change in response to permuting feature position. We establish a game-theoretic connection between OrdShap and Sanchez-Berganti\~nos values, providing a theoretically grounded approach to position-sensitive attribution. Empirical results from health, natural language, and synthetic datasets highlight OrdShap's effectiveness in capturing feature value and feature position attributions, and provide deeper insight into model behavior.