Attention on Multiword Expressions: A Multilingual Study of BERT-based Models with Regard to Idiomaticity and Microsyntax

TL;DR

This study investigates how fine-tuning BERT-based models affects their attention to multiword expressions, specifically idioms and microsyntactic units, across six Indo-European languages, revealing task-dependent attention patterns.

Contribution

It provides a detailed analysis of attention shifts in BERT models for MWEs after fine-tuning on semantic and syntactic tasks, highlighting differences across languages and expression types.

Findings

Fine-tuning influences attention distribution to MWEs.

Semantic tasks lead to more even attention to idioms.

Syntactic tasks increase attention to MSUs in lower layers.

Abstract

This study analyzes the attention patterns of fine-tuned encoder-only models based on the BERT architecture (BERT-based models) towards two distinct types of Multiword Expressions (MWEs): idioms and microsyntactic units (MSUs). Idioms present challenges in semantic non-compositionality, whereas MSUs demonstrate unconventional syntactic behavior that does not conform to standard grammatical categorizations. We aim to understand whether fine-tuning BERT-based models on specific tasks influences their attention to MWEs, and how this attention differs between semantic and syntactic tasks. We examine attention scores to MWEs in both pre-trained and fine-tuned BERT-based models. We utilize monolingual models and datasets in six Indo-European languages - English, German, Dutch, Polish, Russian, and Ukrainian. Our results show that fine-tuning significantly influences how models allocate attention to MWEs. Specifically, models fine-tuned on semantic tasks tend to distribute attention to idiomatic expressions more evenly across layers. Models fine-tuned on syntactic tasks show an increase in attention to MSUs in the lower layers, corresponding with syntactic processing requirements.