Towards understanding evolution of science through language model series

TL;DR

AnnualBERT is a series of language models trained on scientific texts over time, capturing the evolution of scientific language and discourse, and achieving state-of-the-art results in domain-specific NLP tasks.

Contribution

The paper introduces AnnualBERT, a novel series of models trained on full-text scientific papers annually, emphasizing temporal evolution and word-level tokenization, differing from existing subword-based models.

Findings

AnnualBERT models perform comparably to standard models on general tasks.

They achieve state-of-the-art results on scientific NLP and citation link prediction.

Probing reveals insights into scientific language development over time.

Abstract

We introduce AnnualBERT, a series of language models designed specifically to capture the temporal evolution of scientific text. Deviating from the prevailing paradigms of subword tokenizations and "one model to rule them all", AnnualBERT adopts whole words as tokens and is composed of a base RoBERTa model pretrained from scratch on the full-text of 1.7 million arXiv papers published until 2008 and a collection of progressively trained models on arXiv papers at an annual basis. We demonstrate the effectiveness of AnnualBERT models by showing that they not only have comparable performances in standard tasks but also achieve state-of-the-art performances on domain-specific NLP tasks as well as link prediction tasks in the arXiv citation network. We then utilize probing tasks to quantify the models' behavior in terms of representation learning and forgetting as time progresses. Our approach enables the pretrained models to not only improve performances on scientific text processing tasks but also to provide insights into the development of scientific discourse over time. The series of the models is available at https://huggingface.co/jd445/AnnualBERTs.