Locally Aggregated Feature Attribution on Natural Language Model Understanding

TL;DR

This paper introduces LAFA, a gradient-based feature attribution method for NLP that aggregates similar reference texts using language model embeddings, improving interpretability without relying on explicit reference tokens.

Contribution

LAFA is a novel gradient-based attribution method for NLP that avoids obscure reference tokens by aggregating similar texts through language model embeddings.

Findings

LAFA outperforms existing attribution methods on NLP tasks.

It effectively identifies key features in sentiment analysis and entity recognition.

The method demonstrates superior interpretability on multiple datasets.

Abstract

With the growing popularity of deep-learning models, model understanding becomes more important. Much effort has been devoted to demystify deep neural networks for better interpretability. Some feature attribution methods have shown promising results in computer vision, especially the gradient-based methods where effectively smoothing the gradients with reference data is key to a robust and faithful result. However, direct application of these gradient-based methods to NLP tasks is not trivial due to the fact that the input consists of discrete tokens and the "reference" tokens are not explicitly defined. In this work, we propose Locally Aggregated Feature Attribution (LAFA), a novel gradient-based feature attribution method for NLP models. Instead of relying on obscure reference tokens, it smooths gradients by aggregating similar reference texts derived from language model embeddings. For evaluation purpose, we also design experiments on different NLP tasks including Entity Recognition and Sentiment Analysis on public datasets as well as key feature detection on a constructed Amazon catalogue dataset. The superior performance of the proposed method is demonstrated through experiments.