Probing BERT for German Compound Semantics

TL;DR

This study examines how well pretrained German BERT models understand noun compound semantics, revealing that they encode some compositionality information mainly in early layers, but perform worse than English models likely due to linguistic complexity.

Contribution

The paper provides a comprehensive analysis of German BERT's encoding of noun compound semantics, highlighting differences from English models and the impact of German linguistic features.

Findings

Compositionality information is most accessible in early layers.

German BERT models lag behind English models in encoding compound semantics.

Higher productivity and ambiguity in German compounds increase task difficulty.

Abstract

This paper investigates the extent to which pretrained German BERT encodes knowledge of noun compound semantics. We comprehensively vary combinations of target tokens, layers, and cased vs. uncased models, and evaluate them by predicting the compositionality of 868 gold standard compounds. Looking at representational patterns within the transformer architecture, we observe trends comparable to equivalent prior work on English, with compositionality information most easily recoverable in the early layers. However, our strongest results clearly lag behind those reported for English, suggesting an inherently more difficult task in German. This may be due to the higher productivity of compounding in German than in English and the associated increase in constituent-level ambiguity, including in our target compound set.