BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP

TL;DR

BioClinical ModernBERT is a new encoder model for biomedical and clinical NLP that leverages long-context processing, extensive domain-specific pretraining on a large corpus, and diverse datasets to improve performance and speed.

Contribution

It introduces a domain-adapted encoder built on ModernBERT with long-context capabilities, trained on the largest biomedical corpus and diverse datasets, surpassing existing models in multiple tasks.

Findings

Outperforms existing biomedical and clinical encoders on four downstream tasks.

Built on the largest biomedical corpus with over 53.5 billion tokens.

Available in base and large versions with training checkpoints.

Abstract

Encoder-based transformer models are central to biomedical and clinical Natural Language Processing (NLP), as their bidirectional self-attention makes them well-suited for efficiently extracting structured information from unstructured text through discriminative tasks. However, encoders have seen slower development compared to decoder models, leading to limited domain adaptation in biomedical and clinical settings. We introduce BioClinical ModernBERT, a domain-adapted encoder that builds on the recent ModernBERT release, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is developed through continued pretraining on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. It outperforms existing biomedical and clinical encoders on four downstream tasks spanning a broad range of use cases. We release both base (150M parameters) and large (396M parameters) versions of BioClinical ModernBERT, along with training checkpoints to support further research.