LISA: Explaining Recurrent Neural Network Judgments via Layer-wIse Semantic Accumulation and Example to Pattern Transformation

TL;DR

This paper introduces LISA, a technique for interpreting RNN decisions by analyzing how they accumulate semantics over sequences and identifying salient patterns, enhancing understanding of their decision-making process in NLP tasks.

Contribution

The paper proposes LISA, a novel layer-wise semantic accumulation method, to explain RNN judgments and visualize saliency patterns in natural language processing.

Findings

LISA reveals how RNNs build semantics during processing.

Saliency patterns differ across categories and inputs.

The approach improves interpretability of RNN decisions.

Abstract

Recurrent neural networks (RNNs) are temporal networks and cumulative in nature that have shown promising results in various natural language processing tasks. Despite their success, it still remains a challenge to understand their hidden behavior. In this work, we analyze and interpret the cumulative nature of RNN via a proposed technique named as Layer-wIse-Semantic-Accumulation (LISA) for explaining decisions and detecting the most likely (i.e., saliency) patterns that the network relies on while decision making. We demonstrate (1) LISA: "How an RNN accumulates or builds semantics during its sequential processing for a given text example and expected response" (2) Example2pattern: "How the saliency patterns look like for each category in the data according to the network in decision making". We analyse the sensitiveness of RNNs about different inputs to check the increase or decrease in prediction scores and further extract the saliency patterns learned by the network. We employ two relation classification datasets: SemEval 10 Task 8 and TAC KBP Slot Filling to explain RNN predictions via the LISA and example2pattern.