Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Models

TL;DR

This paper introduces influence paths, a causal method to analyze how LSTM language models learn and process subject-verb number agreement, providing detailed insights into their structural understanding of language.

Contribution

The paper presents influence paths as a novel causal framework to interpret LSTM's internal mechanisms for subject-verb agreement, surpassing prior diagnostic methods.

Findings

Influence paths localize and segment the concept of subject-verb agreement.

The method reveals how LSTMs handle attractors and interfering elements.

Results show a more complete understanding of LSTM's structural language processing.

Abstract

LSTM-based recurrent neural networks are the state-of-the-art for many natural language processing (NLP) tasks. Despite their performance, it is unclear whether, or how, LSTMs learn structural features of natural languages such as subject-verb number agreement in English. Lacking this understanding, the generality of LSTM performance on this task and their suitability for related tasks remains uncertain. Further, errors cannot be properly attributed to a lack of structural capability, training data omissions, or other exceptional faults. We introduce *influence paths*, a causal account of structural properties as carried by paths across gates and neurons of a recurrent neural network. The approach refines the notion of influence (the subject's grammatical number has influence on the grammatical number of the subsequent verb) into a set of gate or neuron-level paths. The set localizes and segments the concept (e.g., subject-verb agreement), its constituent elements (e.g., the subject), and related or interfering elements (e.g., attractors). We exemplify the methodology on a widely-studied multi-layer LSTM language model, demonstrating its accounting for subject-verb number agreement. The results offer both a finer and a more complete view of an LSTM's handling of this structural aspect of the English language than prior results based on diagnostic classifiers and ablation.