A Comparative Study of Transformers on Word Sense Disambiguation
Avi Chawla, Nidhi Mulay, Vikas Bishnoi, Gaurav Dhama, Anil, Kumar Singh

TL;DR
This paper compares nine Transformer models to evaluate their effectiveness in Word Sense Disambiguation tasks, revealing insights into their contextualization capabilities and achieving state-of-the-art results.
Contribution
It provides a comprehensive comparative analysis of popular Transformer models on WSD tasks, highlighting their relative strengths and weaknesses.
Findings
Transformer models vary significantly in WSD performance
The proposed kNN approach on CWEs outperforms existing methods
Certain models achieve state-of-the-art results on SensEval datasets
Abstract
Recent years of research in Natural Language Processing (NLP) have witnessed dramatic growth in training large models for generating context-aware language representations. In this regard, numerous NLP systems have leveraged the power of neural network-based architectures to incorporate sense information in embeddings, resulting in Contextualized Word Embeddings (CWEs). Despite this progress, the NLP community has not witnessed any significant work performing a comparative study on the contextualization power of such architectures. This paper presents a comparative study and an extensive analysis of nine widely adopted Transformer models. These models are BERT, CTRL, DistilBERT, OpenAI-GPT, OpenAI-GPT2, Transformer-XL, XLNet, ELECTRA, and ALBERT. We evaluate their contextualization power using two lexical sample Word Sense Disambiguation (WSD) tasks, SensEval-2 and SensEval-3. We adopt…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNatural Language Processing Techniques · Topic Modeling · Speech and dialogue systems
MethodsMulti-Head Attention · Attention Is All You Need · Linear Layer · LAMB · WordPiece · Byte Pair Encoding · Label Smoothing · Weight Decay · Absolute Position Encodings · Adaptive Softmax
