A Hierarchical and Attentional Analysis of Argument Structure Constructions in BERT Using Naturalistic Corpora
Liu Kaipeng, Wu Ling

TL;DR
This paper explores how BERT processes argument structure constructions using a multi-dimensional analysis framework, revealing a hierarchical structure where construction-specific information emerges and is maintained across layers.
Contribution
It introduces a comprehensive analytical approach combining multiple techniques to understand BERT's internal representations of argument structures.
Findings
Hierarchical organization of argument structure information in BERT layers
Construction-specific information appears early and remains through layers
Clusters of argument structures are maximally separable in middle layers
Abstract
This study investigates how the Bidirectional Encoder Representations from Transformers model processes four fundamental Argument Structure Constructions. We employ a multi-dimensional analytical framework, which integrates MDS, t-SNE as dimensionality reduction, Generalized Discrimination Value (GDV) as cluster separation metrics, Fisher Discriminant Ratio (FDR) as linear diagnostic probing, and attention mechanism analysis. Our results reveal a hierarchical representational structure. Construction-specific information emerges in early layers, forms maximally separable clusters in middle layers, and is maintained through later processing stages.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeurobiology of Language and Bilingualism · Embodied and Extended Cognition · Language, Metaphor, and Cognition
