Using Two Losses and Two Datasets Simultaneously to Improve TempoWiC Accuracy

TL;DR

This paper enhances Word Sense Disambiguation for temporal language understanding by training RoBERTa models with dual losses and datasets, achieving a significant accuracy improvement over previous baselines.

Contribution

The study introduces a novel training approach using two losses and datasets simultaneously to improve TempoWiC accuracy.

Findings

Achieved 74.56% macro-F1, surpassing the baseline by 4.23%.

Demonstrated the effectiveness of dual-loss training for temporal WSD.

Showed improved generalization with additional dataset use.

Abstract

WSD (Word Sense Disambiguation) is the task of identifying which sense of a word is meant in a sentence or other segment of text. Researchers have worked on this task (e.g. Pustejovsky, 2002) for years but it's still a challenging one even for SOTA (state-of-the-art) LMs (language models). The new dataset, TempoWiC introduced by Loureiro et al. (2022b) focuses on the fact that words change over time. Their best baseline achieves 70.33% macro-F1. In this work, we use two different losses simultaneously to train RoBERTa-based classification models. We also improve our model by using another similar dataset to generalize better. Our best configuration beats their best baseline by 4.23% and reaches 74.56% macroF1.