Towards Building Multilingual Language Model for Medicine

TL;DR

This paper introduces a large-scale multilingual medical corpus, a benchmark, and a new model that significantly advances multilingual medical language understanding, rivaling even GPT-4 in performance.

Contribution

The paper presents a comprehensive multilingual medical corpus, a benchmark for evaluation, and a new 8B parameter model that outperforms existing open-source models in medical NLP tasks.

Findings

The MMed-Llama 3 model outperforms other open-source models on the benchmark.

The large-scale corpus enables effective domain adaptation for multilingual medical LLMs.

The model rivals GPT-4 in performance despite having fewer parameters.

Abstract

The development of open-source, multilingual medical language models can benefit a wide, linguistically diverse audience from different regions. To promote this domain, we present contributions from the following: First, we construct a multilingual medical corpus, containing approximately 25.5B tokens encompassing 6 main languages, termed as MMedC, enabling auto-regressive domain adaptation for general LLMs; Second, to monitor the development of multilingual medical LLMs, we propose a multilingual medical multi-choice question-answering benchmark with rationale, termed as MMedBench; Third, we have assessed a number of open-source large language models (LLMs) on our benchmark, along with those further auto-regressive trained on MMedC. Our final model, MMed-Llama 3, with only 8B parameters, achieves superior performance compared to all other open-source models on both MMedBench and English benchmarks, even rivaling GPT-4. In conclusion, in this work, we present a large-scale corpus, a benchmark and a series of models to support the development of multilingual medical LLMs.