Probing BERT for German Compound Semantics
Filip Mileti\'c, Aaron Schmid, Sabine Schulte im Walde

TL;DR
This study examines how well pretrained German BERT models understand noun compound semantics, revealing that they encode some compositionality information mainly in early layers, but perform worse than English models likely due to linguistic complexity.
Contribution
The paper provides a comprehensive analysis of German BERT's encoding of noun compound semantics, highlighting differences from English models and the impact of German linguistic features.
Findings
Compositionality information is most accessible in early layers.
German BERT models lag behind English models in encoding compound semantics.
Higher productivity and ambiguity in German compounds increase task difficulty.
Abstract
This paper investigates the extent to which pretrained German BERT encodes knowledge of noun compound semantics. We comprehensively vary combinations of target tokens, layers, and cased vs. uncased models, and evaluate them by predicting the compositionality of 868 gold standard compounds. Looking at representational patterns within the transformer architecture, we observe trends comparable to equivalent prior work on English, with compositionality information most easily recoverable in the early layers. However, our strongest results clearly lag behind those reported for English, suggesting an inherently more difficult task in German. This may be due to the higher productivity of compounding in German than in English and the associated increase in constituent-level ambiguity, including in our target compound set.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Semantic Web and Ontologies
MethodsLinear Warmup With Linear Decay · Softmax · Attention Dropout · WordPiece · Refunds@Expedia|||How do I get a full refund from Expedia? · Attention Is All You Need · Linear Layer · Residual Connection · Weight Decay · Dropout
