Attention Can Reflect Syntactic Structure (If You Let It)
Vinit Ravishankar, Artur Kulmizev, Mostafa Abdou, Anders S{\o}gaard,, Joakim Nivre

TL;DR
This paper investigates whether attention mechanisms in multilingual BERT encode syntactic structures across 18 languages, showing that dependency trees can be decoded from attention patterns and that fine-tuning influences this encoding.
Contribution
It demonstrates that attention heads in multilingual BERT encode syntactic dependency structures across diverse languages and explores how fine-tuning affects this encoding.
Findings
Full dependency trees can be decoded from attention heads.
Individual syntactic relations are tracked by consistent heads across languages.
Fine-tuning on parsing objectives preserves and modifies attention-based structural representations.
Abstract
Since the popularization of the Transformer as a general-purpose feature encoder for NLP, many studies have attempted to decode linguistic structure from its novel multi-head attention mechanism. However, much of such work focused almost exclusively on English -- a language with rigid word order and a lack of inflectional morphology. In this study, we present decoding experiments for multilingual BERT across 18 languages in order to test the generalizability of the claim that dependency syntax is reflected in attention patterns. We show that full trees can be decoded above baseline accuracy from single attention heads, and that individual relations are often tracked by the same heads across languages. Furthermore, in an attempt to address recent debates about the status of attention as an explanatory mechanism, we experiment with fine-tuning mBERT on a supervised parsing objective while…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
MethodsLinear Layer · mBERT · Absolute Position Encodings · Position-Wise Feed-Forward Layer · Layer Normalization · Refunds@Expedia|||How do I get a full refund from Expedia? · Residual Connection · WordPiece · Attention Dropout · Label Smoothing
