Distilling Monolingual and Crosslingual Word-in-Context Representations

TL;DR

This paper introduces a method to distill context-aware word representations from pre-trained language models without needing additional annotated data or model fine-tuning, improving monolingual and crosslingual semantic tasks.

Contribution

It presents a novel approach that combines hidden layer outputs via self-attention to create effective word-in-context representations without modifying the original model.

Findings

Competitive performance on monolingual lexical semantic tasks

Outperformed previous methods in semantic textual similarity estimation

Significant improvements in crosslingual word representations

Abstract

In this study, we propose a method that distils representations of word meaning in context from a pre-trained masked language model in both monolingual and crosslingual settings. Word representations are the basis for context-aware lexical semantics and unsupervised semantic textual similarity (STS) estimation. Different from existing approaches, our method does not require human-annotated corpora nor updates of the parameters of the pre-trained model. The latter feature is appealing for practical scenarios where the off-the-shelf pre-trained model is a common asset among different applications. Specifically, our method learns to combine the outputs of different hidden layers of the pre-trained model using self-attention. Our auto-encoder based training only requires an automatically generated corpus. To evaluate the performance of the proposed approach, we performed extensive experiments using various benchmark tasks. The results on the monolingual tasks confirmed that our representations exhibited a competitive performance compared to that of the previous study for the context-aware lexical semantic tasks and outperformed it for STS estimation. The results of the crosslingual tasks revealed that the proposed method largely improved crosslingual word representations of multilingual pre-trained models.