A Comparative Study of Transformers on Word Sense Disambiguation

TL;DR

This paper compares nine Transformer models to evaluate their effectiveness in Word Sense Disambiguation tasks, revealing insights into their contextualization capabilities and achieving state-of-the-art results.

Contribution

It provides a comprehensive comparative analysis of popular Transformer models on WSD tasks, highlighting their relative strengths and weaknesses.

Findings

Transformer models vary significantly in WSD performance

The proposed kNN approach on CWEs outperforms existing methods

Certain models achieve state-of-the-art results on SensEval datasets

Abstract

Recent years of research in Natural Language Processing (NLP) have witnessed dramatic growth in training large models for generating context-aware language representations. In this regard, numerous NLP systems have leveraged the power of neural network-based architectures to incorporate sense information in embeddings, resulting in Contextualized Word Embeddings (CWEs). Despite this progress, the NLP community has not witnessed any significant work performing a comparative study on the contextualization power of such architectures. This paper presents a comparative study and an extensive analysis of nine widely adopted Transformer models. These models are BERT, CTRL, DistilBERT, OpenAI-GPT, OpenAI-GPT2, Transformer-XL, XLNet, ELECTRA, and ALBERT. We evaluate their contextualization power using two lexical sample Word Sense Disambiguation (WSD) tasks, SensEval-2 and SensEval-3. We adopt a simple yet effective approach to WSD that uses a k-Nearest Neighbor (kNN) classification on CWEs. Experimental results show that the proposed techniques also achieve superior results over the current state-of-the-art on both the WSD tasks