Cross-lingual Word Sense Disambiguation using mBERT Embeddings with Syntactic Dependencies

TL;DR

This paper explores enhancing cross-lingual WSD by integrating syntactic dependency information into mBERT embeddings, aiming to improve disambiguation accuracy across languages.

Contribution

It introduces a method to incorporate syntactic dependencies into mBERT embeddings and proposes techniques to reduce embedding size for better WSD performance.

Findings

High dimensionality of syntax-incorporated embeddings poses challenges

Dependency-based embeddings encode useful syntactic relations

Size reduction methods impact embedding effectiveness

Abstract

Cross-lingual word sense disambiguation (WSD) tackles the challenge of disambiguating ambiguous words across languages given context. The pre-trained BERT embedding model has been proven to be effective in extracting contextual information of words, and have been incorporated as features into many state-of-the-art WSD systems. In order to investigate how syntactic information can be added into the BERT embeddings to result in both semantics- and syntax-incorporated word embeddings, this project proposes the concatenated embeddings by producing dependency parse tress and encoding the relative relationships of words into the input embeddings. Two methods are also proposed to reduce the size of the concatenated embeddings. The experimental results show that the high dimensionality of the syntax-incorporated embeddings constitute an obstacle for the classification task, which needs to be further addressed in future studies.