What Does Neuro Mean to Cardio? Investigating the Role of Clinical Specialty Data in Medical LLMs

TL;DR

This paper introduces S-MedQA, a large medical QA dataset across 15 specialties, and investigates how clinical specialty data influences LLM performance, revealing that domain shift may be more impactful than specialty-specific fine-tuning.

Contribution

The creation of S-MedQA, a comprehensive dataset for benchmarking medical LLMs across specialties, and an analysis of the effects of specialty data on model performance and knowledge representation.

Findings

Training on a specialty does not guarantee better performance in that specialty.

Token probabilities of relevant terms increase across all specialties regardless of training data.

Domain shifting from general to medical domains has a significant impact on model improvements.

Abstract

In this paper, we introduce S-MedQA, an English medical question-answering (QA) dataset designed for benchmarking large language models (LLMs) in fine-grained clinical specialties. S-MedQA consists of over 24k examples, covering 15 medical specialties, with QA pairs that can have multiple specialty annotations, such as when a question is cross-disciplinary. The dataset is constructed using both machine and expert verification to maximize data availability and reliability. We use S-MedQA to investigate the role of clinical specialties in the knowledge-intensive scenario of medical QA. Our results show that training on data from a clinical specialty does not necessarily lead to the best performance on that specialty. Additionally, regardless of the specialty the LLM was fine-tuned on, token probabilities of clinically relevant terms consistently increase across all specialties. Based on these findings, we hypothesize that improvement gains, at least in our settings, are derived primarily from domain shifting (e.g., general to medical) rather than from injecting specialty-specific knowledge. This suggests a need to rethink the role of fine-tuning data in the medical domain. To encourage further advancements in the clinical NLP field, we release S-MedQA along with all the code required to reproduce our experiments for the research community.