Progressive Inference: Explaining Decoder-Only Sequence Classification Models Using Intermediate Predictions

TL;DR

This paper introduces Progressive Inference, a framework that explains decoder-only sequence classification models by leveraging intermediate predictions at different input points, enabling more accurate attributions with minimal extra computation.

Contribution

It presents two novel methods, SP-PI and MP-PI, for computing input attributions using intermediate model predictions, improving interpretability of decoder-only models.

Findings

SP-PI and MP-PI outperform prior attribution methods.

Intermediate predictions enable efficient, fine-grained explanations.

Methods are validated on diverse text classification models.

Abstract

This paper proposes Progressive Inference - a framework to compute input attributions to explain the predictions of decoder-only sequence classification models. Our work is based on the insight that the classification head of a decoder-only Transformer model can be used to make intermediate predictions by evaluating them at different points in the input sequence. Due to the causal attention mechanism, these intermediate predictions only depend on the tokens seen before the inference point, allowing us to obtain the model's prediction on a masked input sub-sequence, with negligible computational overheads. We develop two methods to provide sub-sequence level attributions using this insight. First, we propose Single Pass-Progressive Inference (SP-PI), which computes attributions by taking the difference between consecutive intermediate predictions. Second, we exploit a connection with Kernel SHAP to develop Multi Pass-Progressive Inference (MP-PI). MP-PI uses intermediate predictions from multiple masked versions of the input to compute higher quality attributions. Our studies on a diverse set of models trained on text classification tasks show that SP-PI and MP-PI provide significantly better attributions compared to prior work.