BioMamba: Domain-Adaptive Biomedical Language Models

TL;DR

BioMamba is a family of biomedical language models that improves performance on biomedical and clinical tasks while maintaining general language ability through balanced domain-adaptive pretraining.

Contribution

We introduce BioMamba, a novel biomedical language model family that balances domain-specific adaptation with general language preservation using a mixed pretraining approach.

Findings

BioMamba improves PubMed and Wikipedia modeling without degrading C4 performance.

BioMamba transfers effectively to biomedical and clinical tasks with strong downstream results.

The best model achieves PubMed perplexity of 5.28 and high accuracy on BioASQ and PubMedQA.

Abstract

Background: Biomedical language models should improve performance on biomedical text while retaining general-domain language ability. For Mamba-based models, this trade-off has not been clearly studied across biomedical literature and clinical text. Methods: We developed BioMamba, a family of biomedical models obtained by continued pretraining of public Mamba2 checkpoints on PubMed, with small amounts of general-domain data from the Colossal Clean Crawled Corpus (C4) and Wikipedia included to help preserve general-domain language ability. We evaluated language modeling and three downstream tasks across multiple model scales: clinical note completion, discharge summary generation, and biomedical yes/no question answering. Results: BioMamba consistently improved PubMed modeling, improved Wikipedia modeling, and left C4 performance largely unchanged. After supervised fine-tuning, BioMamba transferred well to both biomedical literature and clinical text, yielding strong results on completion, summarization, and question answering. On MIMIC-IV, BioMamba+SFT consistently matched or exceeded SFT from the corresponding base checkpoints across note completion and discharge summary generation. The strongest model achieved a PubMed perplexity of 5.28 and accuracies of 90.24% and 73.00% on BioASQ and PubMedQA, respectively. Conclusion: Balanced domain-adaptive pretraining strategy strengthens Mamba language models for both biomedical literature and clinical text, while preserving general-domain language capabilities, establishing BioMamba as a practical foundation for biomedical NLP applications.