Which *BERT? A Survey Organizing Contextualized Encoders

TL;DR

This survey reviews recent advancements in pretrained contextualized text encoders, organizing shared lessons and themes to aid understanding and model selection in NLP.

Contribution

It consolidates diverse recent efforts into an organized framework, highlighting key considerations for interpreting and choosing language models.

Findings

Identified common themes across recent models

Provided a structured overview of advancements

Highlighted considerations for model interpretation

Abstract

Pretrained contextualized text encoders are now a staple of the NLP community. We present a survey on language representation learning with the aim of consolidating a series of shared lessons learned across a variety of recent efforts. While significant advancements continue at a rapid pace, we find that enough has now been discovered, in different directions, that we can begin to organize advances according to common themes. Through this organization, we highlight important considerations when interpreting recent contributions and choosing which model to use.