Inserting Information Bottlenecks for Attribution in Transformers

TL;DR

This paper introduces a method using information bottlenecks to analyze feature attribution in transformer models like BERT, providing insights into information flow and outperforming existing attribution methods.

Contribution

The paper proposes a novel application of information bottlenecks for feature attribution in transformers, enhancing interpretability and outperforming competing methods.

Findings

Effective attribution of features in BERT

Outperforms two competitive methods in degradation tests

Provides insights into information flow across layers

Abstract

Pretrained transformers achieve the state of the art across tasks in natural language processing, motivating researchers to investigate their inner mechanisms. One common direction is to understand what features are important for prediction. In this paper, we apply information bottlenecks to analyze the attribution of each feature for prediction on a black-box model. We use BERT as the example and evaluate our approach both quantitatively and qualitatively. We show the effectiveness of our method in terms of attribution and the ability to provide insight into how information flows through layers. We demonstrate that our technique outperforms two competitive methods in degradation tests on four datasets. Code is available at https://github.com/bazingagin/IBA.