PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications

TL;DR

PediatricsGPT is a specialized Chinese language model designed for pediatric medical assistance, trained on a large, high-quality dataset with a novel training pipeline, outperforming previous models in pediatric healthcare tasks.

Contribution

This paper introduces PedCorpus, a large pediatric instruction dataset, and PediatricsGPT, a new Chinese pediatric LLM with a systematic training pipeline and innovative fine-tuning strategies.

Findings

PediatricsGPT outperforms previous Chinese medical LLMs in multiple evaluations.

The hybrid instruction pre-training improves domain adaptation.

The mixture of experts strategy enhances pediatric expertise.

Abstract

Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures. To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300,000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline. In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase, a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct doctor downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. Our model and dataset will be open-source for community development.