Analysis and Evaluation of Language Models for Word Sense Disambiguation

TL;DR

This paper provides a comprehensive analysis of BERT's ability to perform Word Sense Disambiguation, highlighting its strengths in capturing sense distinctions and comparing fine-tuning versus feature extraction strategies under limited data conditions.

Contribution

It offers the first detailed evaluation of BERT's WSD capabilities, comparing different strategies and analyzing practical challenges in real-world scenarios.

Findings

BERT accurately captures high-level sense distinctions with limited data.

Feature extraction outperforms fine-tuning in robustness against sense bias.

Averaging contextualized embeddings is effective with as few as three training sentences.

Abstract

Transformer-based language models have taken many fields in NLP by storm. BERT and its derivatives dominate most of the existing evaluation benchmarks, including those for Word Sense Disambiguation (WSD), thanks to their ability in capturing context-sensitive semantic nuances. However, there is still little knowledge about their capabilities and potential limitations in encoding and recovering word senses. In this article, we provide an in-depth quantitative and qualitative analysis of the celebrated BERT model with respect to lexical ambiguity. One of the main conclusions of our analysis is that BERT can accurately capture high-level sense distinctions, even when a limited number of examples is available for each word sense. Our analysis also reveals that in some cases language models come close to solving coarse-grained noun disambiguation under ideal conditions in terms of availability of training data and computing resources. However, this scenario rarely occurs in real-world settings and, hence, many practical challenges remain even in the coarse-grained setting. We also perform an in-depth comparison of the two main language model based WSD strategies, i.e., fine-tuning and feature extraction, finding that the latter approach is more robust with respect to sense bias and it can better exploit limited available training data. In fact, the simple feature extraction strategy of averaging contextualized embeddings proves robust even using only three training sentences per word sense, with minimal improvements obtained by increasing the size of this training data.