An Attribution Method for Siamese Encoders

TL;DR

This paper introduces a novel attribution method for Siamese encoders that explains model predictions by attributing importance to pairs of input features, enabling interpretability of models like sentence transformers.

Contribution

It generalizes integrated gradients to models with multiple inputs, providing a formal, convergent attribution method for Siamese encoders that highlights influential feature pairs.

Findings

Few token-pairs often explain large prediction fractions.

The method highlights nouns and verbs as important features.

Accurate predictions require attending to most input tokens.

Abstract

Despite the success of Siamese encoder models such as sentence transformers (ST), little is known about the aspects of inputs they pay attention to. A barrier is that their predictions cannot be attributed to individual features, as they compare two inputs rather than processing a single one. This paper derives a local attribution method for Siamese encoders by generalizing the principle of integrated gradients to models with multiple inputs. The solution takes the form of feature-pair attributions, and can be reduced to a token-token matrix for STs. Our method involves the introduction of integrated Jacobians and inherits the advantageous formal properties of integrated gradients: it accounts for the model's full computation graph and is guaranteed to converge to the actual prediction. A pilot study shows that in an ST few token-pairs can often explain large fractions of predictions, and it focuses on nouns and verbs. For accurate predictions, it however needs to attend to the majority of tokens and parts of speech.