SciBERT: A Pretrained Language Model for Scientific Text

TL;DR

SciBERT is a specialized pretrained language model based on BERT, designed to improve NLP performance on scientific texts by leveraging large-scale scientific corpora for unsupervised pretraining.

Contribution

It introduces SciBERT, a domain-specific language model trained on scientific literature, achieving superior results on scientific NLP tasks compared to general models like BERT.

Findings

SciBERT outperforms BERT on multiple scientific NLP tasks.

Achieves state-of-the-art results on several scientific datasets.

Demonstrates the effectiveness of domain-specific pretraining.

Abstract

Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SciBERT, a pretrained language model based on BERT (Devlin et al., 2018) to address the lack of high-quality, large-scale labeled scientific data. SciBERT leverages unsupervised pretraining on a large multi-domain corpus of scientific publications to improve performance on downstream scientific NLP tasks. We evaluate on a suite of tasks including sequence tagging, sentence classification and dependency parsing, with datasets from a variety of scientific domains. We demonstrate statistically significant improvements over BERT and achieve new state-of-the-art results on several of these tasks. The code and pretrained models are available at https://github.com/allenai/scibert/.