Obtaining Better Static Word Embeddings Using Contextual Embedding Models

TL;DR

This paper introduces a distillation method that enhances static word embeddings by leveraging contextual models, achieving better quality and efficiency, and enabling fair comparison between static and contextual embeddings.

Contribution

The paper presents a simple extension of CBOW training for distillation that improves static embeddings using contextual models, balancing efficiency and quality.

Findings

Distilled static embeddings outperform previous static and distilled embeddings.

The method improves computational efficiency in NLP applications.

Enables fair comparison between static and contextual embeddings.

Abstract

The advent of contextual word embeddings -- representations of words which incorporate semantic and syntactic information from their context -- has led to tremendous improvements on a wide variety of NLP tasks. However, recent contextual models have prohibitively high computational cost in many use-cases and are often hard to interpret. In this work, we demonstrate that our proposed distillation method, which is a simple extension of CBOW-based training, allows to significantly improve computational efficiency of NLP applications, while outperforming the quality of existing static embeddings trained from scratch as well as those distilled from previously proposed methods. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings via standard lexical evaluation tasks.