Exploring the Role of BERT Token Representations to Explain Sentence Probing Results

TL;DR

This paper investigates how BERT encodes linguistic features in token representations, revealing that specific token subspaces are responsible for capturing syntactic and semantic properties, which explains probing results.

Contribution

It introduces a detailed analysis of BERT's token representation space, identifying meaningful subspaces responsible for encoding linguistic features, beyond standard classification methods.

Findings

BERT encodes linguistic features in specific token subspaces.

Token representations can explain BERT's ability to detect grammatical abnormalities.

Distinct subspaces encode grammatical number and tense.

Abstract

Several studies have been carried out on revealing linguistic features captured by BERT. This is usually achieved by training a diagnostic classifier on the representations obtained from different layers of BERT. The subsequent classification accuracy is then interpreted as the ability of the model in encoding the corresponding linguistic property. Despite providing insights, these studies have left out the potential role of token representations. In this paper, we provide a more in-depth analysis on the representation space of BERT in search for distinct and meaningful subspaces that can explain the reasons behind these probing results. Based on a set of probing tasks and with the help of attribution methods we show that BERT tends to encode meaningful knowledge in specific token representations (which are often ignored in standard classification setups), allowing the model to detect syntactic and semantic abnormalities, and to distinctively separate grammatical number and tense subspaces.