PMC-LLaMA: Towards Building Open-source Language Models for Medicine

TL;DR

This paper presents PMC-LLaMA, an open-source medical language model built through domain-specific data integration and fine-tuning, achieving superior performance on medical question-answering benchmarks with a 13-billion-parameter model.

Contribution

It introduces a systematic approach for adapting general language models to the medical domain, including data collection, fine-tuning, and evaluation, with a new large-scale medical instruction dataset.

Findings

PMC-LLaMA outperforms existing models on medical QA benchmarks.

The model surpasses ChatGPT in medical question-answering tasks.

A comprehensive dataset of 202M tokens was created for instruction tuning.

Abstract

Recently, Large Language Models (LLMs) have showcased remarkable capabilities in natural language understanding. While demonstrating proficiency in everyday conversations and question-answering situations, these models frequently struggle in domains that require precision, such as medical applications, due to their lack of domain-specific knowledge. In this paper, we describe the procedure for building a powerful, open-source language model specifically designed for medicine applications, termed as PMC-LLaMA. Our contributions are threefold: (i) we systematically investigate the process of adapting a general-purpose foundation language model towards medical domain, this involves data-centric knowledge injection through the integration of 4.8M biomedical academic papers and 30K medical textbooks, as well as comprehensive fine-tuning for alignment with domain-specific instructions; (ii) we contribute a large-scale, comprehensive dataset for instruction tuning. This dataset encompasses medical question-answering (QA), rationale for reasoning, and conversational dialogues, comprising a total of 202M tokens; (iii) we conduct thorough ablation studies to demonstrate the effectiveness of each proposed component. While evaluating on various public medical question-answering benchmarks, our lightweight PMCLLaMA, which consists of only 13 billion parameters, exhibits superior performance, even surpassing ChatGPT. All models, codes, datasets can be found in https://github.com/chaoyi-wu/PMC-LLaMA.