ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD

TL;DR

This paper fine-tunes Arabic BERT models for Word Sense Disambiguation by treating it as a sentence-pair classification task, using a large dataset of context-gloss pairs, achieving promising accuracy.

Contribution

It introduces a new dataset of Arabic context-gloss pairs and demonstrates effective fine-tuning of BERT for Arabic WSD.

Findings

Achieved 84% accuracy on Arabic WSD task

Constructed a dataset of 167,000 labeled pairs

Explored different supervised signals for target word emphasis

Abstract

Using pre-trained transformer models such as BERT has proven to be effective in many NLP tasks. This paper presents our work to fine-tune BERT models for Arabic Word Sense Disambiguation (WSD). We treated the WSD task as a sentence-pair binary classification task. First, we constructed a dataset of labeled Arabic context-gloss pairs (~167k pairs) we extracted from the Arabic Ontology and the large lexicographic database available at Birzeit University. Each pair was labeled as True or False and target words in each context were identified and annotated. Second, we used this dataset for fine-tuning three pre-trained Arabic BERT models. Third, we experimented the use of different supervised signals used to emphasize target words in context. Our experiments achieved promising results (accuracy of 84%) although we used a large set of senses in the experiment.