Baichuan-M1: Pushing the Medical Capability of Large Language Models

TL;DR

Baichuan-M1 is a domain-specific large language model trained from scratch on medical data, achieving strong performance in both general and medical tasks, and is open-sourced for broader use.

Contribution

We developed Baichuan-M1, a medical domain-specific LLM trained from scratch on 20 trillion tokens, focusing on enhancing medical capabilities beyond traditional fine-tuning methods.

Findings

Performs well in general tasks like mathematics and coding

Excels in specialized medical fields

Open-sourced model for community use

Abstract

The current generation of large language models (LLMs) is typically designed for broad, general-purpose applications, while domain-specific LLMs, especially in vertical fields like medicine, remain relatively scarce. In particular, the development of highly efficient and practical LLMs for the medical domain is challenging due to the complexity of medical knowledge and the limited availability of high-quality data. To bridge this gap, we introduce Baichuan-M1, a series of large language models specifically optimized for medical applications. Unlike traditional approaches that simply continue pretraining on existing models or apply post-training to a general base model, Baichuan-M1 is trained from scratch with a dedicated focus on enhancing medical capabilities. Our model is trained on 20 trillion tokens and incorporates a range of effective training methods that strike a balance between general capabilities and medical expertise. As a result, Baichuan-M1 not only performs strongly across general domains such as mathematics and coding but also excels in specialized medical fields. We have open-sourced Baichuan-M1-14B, a mini version of our model, which can be accessed through the following links.