MEDITRON-70B: Scaling Medical Pretraining for Large Language Models

TL;DR

MEDITRON-70B is an open-source large medical language model with 70 billion parameters, trained on extensive medical data, outperforming many existing models and making advanced medical AI more accessible.

Contribution

The paper introduces MEDITRON, a large-scale open-source medical LLM with 7B and 70B parameters, trained on curated medical data, and demonstrates its superior performance over existing models.

Findings

MEDITRON-70B outperforms several state-of-the-art baselines.

It achieves a 6% absolute performance gain over the best public baseline.

MEDITRON-70B surpasses GPT-3.5 and Med-PaLM in medical benchmarks.

Abstract

Large language models (LLMs) can potentially democratize access to medical knowledge. While many efforts have been made to harness and improve LLMs' medical knowledge and reasoning capacities, the resulting models are either closed-source (e.g., PaLM, GPT-4) or limited in scale (<= 13B parameters), which restricts their abilities. In this work, we improve access to large-scale medical LLMs by releasing MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain. MEDITRON builds on Llama-2 (through our adaptation of Nvidia's Megatron-LM distributed trainer), and extends pretraining on a comprehensively curated medical corpus, including selected PubMed articles, abstracts, and internationally-recognized medical guidelines. Evaluations using four major medical benchmarks show significant performance gains over several state-of-the-art baselines before and after task-specific finetuning. Overall, MEDITRON achieves a 6% absolute performance gain over the best public baseline in its parameter class and 3% over the strongest baseline we finetuned from Llama-2. Compared to closed-source LLMs, MEDITRON-70B outperforms GPT-3.5 and Med-PaLM and is within 5% of GPT-4 and 10% of Med-PaLM-2. We release our code for curating the medical pretraining corpus and the MEDITRON model weights to drive open-source development of more capable medical LLMs.