Training Bi-Encoders for Word Sense Disambiguation

TL;DR

This paper enhances bi-encoder models for Word Sense Disambiguation by optimizing training strategies and lexical information presentation, achieving state-of-the-art results through multi-stage pre-training and fine-tuning.

Contribution

It introduces novel training and lexical presentation methods for bi-encoders, advancing the state of the art in WSD.

Findings

Achieved new state-of-the-art WSD performance

Demonstrated effectiveness of multi-stage pre-training

Improved lexical information integration methods

Abstract

Modern transformer-based neural architectures yield impressive results in nearly every NLP task and Word Sense Disambiguation, the problem of discerning the correct sense of a word in a given context, is no exception. State-of-the-art approaches in WSD today leverage lexical information along with pre-trained embeddings from these models to achieve results comparable to human inter-annotator agreement on standard evaluation benchmarks. In the same vein, we experiment with several strategies to optimize bi-encoders for this specific task and propose alternative methods of presenting lexical information to our model. Through our multi-stage pre-training and fine-tuning pipeline we further the state of the art in Word Sense Disambiguation.