Building Interpretable Interaction Trees for Deep NLP Models

TL;DR

This paper introduces a novel method to interpret deep NLP models by constructing interaction trees based on word contributions, providing insights into how models like BERT and LSTM encode word interactions.

Contribution

The paper presents a new approach to disentangle and quantify word interactions in deep NLP models using Shapley values and tree structures, enhancing interpretability.

Findings

Effective in analyzing BERT, ELMo, LSTM, CNN, and Transformer models.

Provides new insights into internal word interaction mechanisms.

Demonstrates the method's usefulness in understanding model predictions.

Abstract

This paper proposes a method to disentangle and quantify interactions among words that are encoded inside a DNN for natural language processing. We construct a tree to encode salient interactions extracted by the DNN. Six metrics are proposed to analyze properties of interactions between constituents in a sentence. The interaction is defined based on Shapley values of words, which are considered as an unbiased estimation of word contributions to the network prediction. Our method is used to quantify word interactions encoded inside the BERT, ELMo, LSTM, CNN, and Transformer networks. Experimental results have provided a new perspective to understand these DNNs, and have demonstrated the effectiveness of our method.