Influence Patterns for Explaining Information Flow in BERT

TL;DR

This paper introduces influence patterns to explain how information flows in BERT, revealing that skip connections play a major role and that pattern consistency correlates with model performance.

Contribution

The paper proposes influence patterns as a novel method to analyze information flow in BERT, highlighting the importance of skip connections and pattern consistency.

Findings

Significant information flow occurs through skip connections.

Pattern consistency correlates with BERT's performance.

Patterns explain more of the model's behavior than previous methods.

Abstract

While attention is all you need may be proving true, we do not know why: attention-based transformer models such as BERT are superior but how information flows from input tokens to output predictions are unclear. We introduce influence patterns, abstractions of sets of paths through a transformer model. Patterns quantify and localize the flow of information to paths passing through a sequence of model nodes. Experimentally, we find that significant portion of information flow in BERT goes through skip connections instead of attention heads. We further show that consistency of patterns across instances is an indicator of BERT's performance. Finally, We demonstrate that patterns account for far more model performance than previous attention-based and layer-based methods.