Linguistically Grounded Analysis of Language Models using Shapley Head Values

TL;DR

This paper uses Shapley Head Values to analyze how BERT and RoBERTa encode morphosyntactic phenomena, revealing that specific attention heads cluster for related linguistic tasks and support the idea of subnetworks aligned with linguistic theory.

Contribution

It introduces a novel application of Shapley Head Values for probing language models, providing new insights into their organization of linguistic information.

Findings

Attention heads for related phenomena cluster together.

SHV attributions reveal distinct patterns across models.

Supports the hypothesis of subnetworks corresponding to linguistic theory.

Abstract

Understanding how linguistic knowledge is encoded in language models is crucial for improving their generalisation capabilities. In this paper, we investigate the processing of morphosyntactic phenomena, by leveraging a recently proposed method for probing language models via Shapley Head Values (SHVs). Using the English language BLiMP dataset, we test our approach on two widely used models, BERT and RoBERTa, and compare how linguistic constructions such as anaphor agreement and filler-gap dependencies are handled. Through quantitative pruning and qualitative clustering analysis, we demonstrate that attention heads responsible for processing related linguistic phenomena cluster together. Our results show that SHV-based attributions reveal distinct patterns across both models, providing insights into how language models organize and process linguistic information. These findings support the hypothesis that language models learn subnetworks corresponding to linguistic theory, with potential implications for cross-linguistic model analysis and interpretability in Natural Language Processing (NLP).