Contextual Embeddings: When Are They Worth It?

TL;DR

This paper evaluates when deep contextual embeddings like BERT outperform simpler embeddings, finding that on large datasets simpler methods often suffice, but contextual models excel with complex language and unseen words.

Contribution

It systematically analyzes the conditions under which contextual embeddings provide significant advantages over traditional and random embeddings.

Findings

Simpler embeddings can match contextual embeddings on large-scale data.

Contextual embeddings outperform in complex, ambiguous, and unseen word scenarios.

Performance gains are most notable with linguistically complex tasks.

Abstract

We study the settings for which deep contextual embeddings (e.g., BERT) give large improvements in performance relative to classic pretrained embeddings (e.g., GloVe), and an even simpler baseline---random word embeddings---focusing on the impact of the training set size and the linguistic properties of the task. Surprisingly, we find that both of these simpler baselines can match contextual embeddings on industry-scale data, and often perform within 5 to 10% accuracy (absolute) on benchmark tasks. Furthermore, we identify properties of data for which contextual embeddings give particularly large gains: language containing complex structure, ambiguous word usage, and words unseen in training.