FEWS: Large-Scale, Low-Shot Word Sense Disambiguation with the Dictionary

TL;DR

FEWS is a large-scale, automatically extracted low-shot WSD dataset from Wiktionary that improves sense coverage and enables better modeling of rare senses, supporting future research in low-resource disambiguation.

Contribution

The paper introduces FEWS, a new low-shot WSD dataset with extensive sense coverage, and demonstrates its utility for training and evaluating models on rare senses.

Findings

Models trained with FEWS better capture rare senses.

Humans outperform baseline models on FEWS.

FEWS provides extensive sense coverage across domains.

Abstract

Current models for Word Sense Disambiguation (WSD) struggle to disambiguate rare senses, despite reaching human performance on global WSD metrics. This stems from a lack of data for both modeling and evaluating rare senses in existing WSD datasets. In this paper, we introduce FEWS (Few-shot Examples of Word Senses), a new low-shot WSD dataset automatically extracted from example sentences in Wiktionary. FEWS has high sense coverage across different natural language domains and provides: (1) a large training set that covers many more senses than previous datasets and (2) a comprehensive evaluation set containing few- and zero-shot examples of a wide variety of senses. We establish baselines on FEWS with knowledge-based and neural WSD approaches and present transfer learning experiments demonstrating that models additionally trained with FEWS better capture rare senses in existing WSD datasets. Finally, we find humans outperform the best baseline models on FEWS, indicating that FEWS will support significant future work on low-shot WSD.