Fine-Tuning In-House Large Language Models to Infer Differential Diagnosis from Radiology Reports

TL;DR

This paper presents a method to fine-tune open-source large language models for extracting differential diagnoses from radiology reports, achieving performance comparable to GPT-4 while improving privacy and reducing costs.

Contribution

The study introduces a pipeline for creating in-house LLMs using GPT-4 generated labels, enabling effective differential diagnosis extraction without relying on proprietary models.

Findings

Fine-tuned LLMs achieved 92.1% F1 score.

Performance comparable to GPT-4 (90.8%).

Method enhances privacy and reduces costs.

Abstract

Radiology reports summarize key findings and differential diagnoses derived from medical imaging examinations. The extraction of differential diagnoses is crucial for downstream tasks, including patient management and treatment planning. However, the unstructured nature of these reports, characterized by diverse linguistic styles and inconsistent formatting, presents significant challenges. Although proprietary large language models (LLMs) such as GPT-4 can effectively retrieve clinical information, their use is limited in practice by high costs and concerns over the privacy of protected health information (PHI). This study introduces a pipeline for developing in-house LLMs tailored to identify differential diagnoses from radiology reports. We first utilize GPT-4 to create 31,056 labeled reports, then fine-tune open source LLM using this dataset. Evaluated on a set of 1,067 reports annotated by clinicians, the proposed model achieves an average F1 score of 92.1\%, which is on par with GPT-4 (90.8\%). Through this study, we provide a methodology for constructing in-house LLMs that: match the performance of GPT, reduce dependence on expensive proprietary models, and enhance the privacy and security of PHI.