Does Biomedical Training Lead to Better Medical Performance?

TL;DR

This study systematically evaluates biomedical LLMs on medical tasks, revealing that biomedical fine-tuning often reduces performance and general models can outperform domain-specific models, highlighting a trade-off in model training.

Contribution

It provides the first comprehensive evaluation of biomedical training effects on medical task performance, revealing potential drawbacks of domain-specific fine-tuning.

Findings

Biomedical fine-tuning often decreases model performance on medical tasks.

General-domain models can outperform biomedical models in medical tasks.

Open-source datasets and scripts facilitate further research.

Abstract

Large Language Models (LLMs) are expected to significantly contribute to patient care, diagnostics, and administrative processes. Emerging biomedical LLMs aim to address healthcare-specific challenges, including privacy demands and computational constraints. Assessing the models' suitability for this sensitive application area is of the utmost importance. However, biomedical training has not been systematically evaluated on medical tasks. This study investigates the effect of biomedical training in the context of six practical medical tasks evaluating $25$ models. In contrast to previous evaluations, our results reveal a performance decline in nine out of twelve biomedical models after fine-tuning, particularly on tasks involving hallucinations, ICD10 coding, and instruction adherence. General-domain models like Meta-Llama-3.1-70B-Instruct outperformed their biomedical counterparts, indicating a trade-off between domain-specific fine-tuning and general medical task performance. We open-source all evaluation scripts and datasets at https://github.com/TIO-IKIM/CLUE to support further research in this critical area.