MirrorWiC: On Eliciting Word-in-Context Representations from Pretrained Language Models

TL;DR

MirrorWiC is an unsupervised fine-tuning method that enhances word-in-context representations in pretrained language models using contrastive learning on raw Wikipedia data, achieving competitive results without labeled sense data.

Contribution

It introduces a simple, fully unsupervised WiC fine-tuning approach that improves context-aware word representations across multiple languages and benchmarks.

Findings

Significant performance improvements over baseline PLMs.

Achieves results comparable to supervised models on some benchmarks.

Effective across monolingual, multilingual, and cross-lingual settings.

Abstract

Recent work indicated that pretrained language models (PLMs) such as BERT and RoBERTa can be transformed into effective sentence and word encoders even via simple self-supervised techniques. Inspired by this line of work, in this paper we propose a fully unsupervised approach to improving word-in-context (WiC) representations in PLMs, achieved via a simple and efficient WiC-targeted fine-tuning procedure: MirrorWiC. The proposed method leverages only raw texts sampled from Wikipedia, assuming no sense-annotated data, and learns context-aware word representations within a standard contrastive learning setup. We experiment with a series of standard and comprehensive WiC benchmarks across multiple languages. Our proposed fully unsupervised MirrorWiC models obtain substantial gains over off-the-shelf PLMs across all monolingual, multilingual and cross-lingual setups. Moreover, on some standard WiC benchmarks, MirrorWiC is even on-par with supervised models fine-tuned with in-task data and sense labels.