How Can BERT Help Lexical Semantics Tasks?

TL;DR

This paper explores how to utilize BERT's dynamic contextualized embeddings to improve static lexical semantics tasks, achieving state-of-the-art results across multiple datasets.

Contribution

It introduces a novel method of using dynamic BERT embeddings to enhance static embeddings for lexical semantics tasks.

Findings

Improved performance on seven lexical semantics datasets

State-of-the-art results achieved with the proposed method

Demonstrates the benefit of combining dynamic and static embeddings

Abstract

Contextualized embeddings such as BERT can serve as strong input representations to NLP tasks, outperforming their static embeddings counterparts such as skip-gram, CBOW and GloVe. However, such embeddings are dynamic, calculated according to a sentence-level context, which limits their use in lexical semantics tasks. We address this issue by making use of dynamic embeddings as word representations in training static embeddings, thereby leveraging their strong representation power for disambiguating context information. Results show that this method leads to improvements over traditional static embeddings on a range of lexical semantics tasks, obtaining the best reported results on seven datasets.