The Limited Impact of Medical Adaptation of Large Language and Vision-Language Models

TL;DR

This study critically evaluates whether domain-specific pretraining of large language and vision-language models genuinely enhances medical question-answering performance, finding limited or inconsistent improvements over base models.

Contribution

It provides a comprehensive comparison of medical versus base models, revealing that domain adaptation often does not lead to significant performance gains in medical tasks.

Findings

Medical models rarely outperform base models in medical QA.

Performance improvements are inconsistent and often statistically insignificant.

General-purpose models may already possess substantial medical knowledge.

Abstract

Several recent works seek to adapt general-purpose large language models (LLMs) and vision-language models (VLMs) for medical applications through continued pretraining on publicly available biomedical corpora. These works typically claim that such domain-adaptive pretraining improves performance on various downstream medical tasks, such as answering medical exam questions. In this paper, we compare ten "medical" LLMs and two VLMs against their corresponding base models, arriving at a different conclusion: all medical VLMs and nearly all medical LLMs fail to consistently improve over their base models in the zero-/few-shot prompting and supervised fine-tuning regimes for medical question answering (QA). For instance, on clinical-note-based QA tasks in the 3-shot setting, medical LLMs outperform their base models in only 26.7% of cases, reach a (statistical) tie in 16.7% of cases, and perform significantly worse in the remaining 56.7% of cases. Our conclusions are based on (i) comparing each medical model directly against its base model; (ii) optimizing the prompts for each model separately in zero-/few-shot prompting; and (iii) accounting for statistical uncertainty in comparisons. Our findings suggest that state-of-the-art general-domain models may already exhibit strong medical knowledge and reasoning capabilities, and offer recommendations to strengthen the conclusions of future studies.