What You Read is What You Classify: Highlighting Attributions to Text and Text-Like Inputs

TL;DR

This paper introduces a mask-based explainable AI method for token classifiers, especially in text and sequence data, which highlights relevant input segments and improves interpretability over existing techniques.

Contribution

It generalizes a mask-based explanation approach from images to token sequences, enabling human-readable explanations for models like transformers.

Findings

Masked segments are less relevant to classification than unmasked ones

The method produces human-readable explanations for token importance

Effective for nucleotide sequence classifiers

Abstract

At present, there are no easily understood explainable artificial intelligence (AI) methods for discrete token inputs, like text. Most explainable AI techniques do not extend well to token sequences, where both local and global features matter, because state-of-the-art models, like transformers, tend to focus on global connections. Therefore, existing explainable AI algorithms fail by (i) identifying disparate tokens of importance, or (ii) assigning a large number of tokens a low value of importance. This method for explainable AI for tokens-based classifiers generalizes a mask-based explainable AI algorithm for images. It starts with an Explainer neural network that is trained to create masks to hide information not relevant for classification. Then, the Hadamard product of the mask and the continuous values of the classifier's embedding layer is taken and passed through the classifier, changing the magnitude of the embedding vector but keeping the orientation unchanged. The Explainer is trained for a taxonomic classifier for nucleotide sequences and it is shown that the masked segments are less relevant to classification than the unmasked ones. This method focused on the importance the token as a whole (i.e., a segment of the input sequence), producing a human-readable explanation.