BURExtract-Llama: An LLM for Clinical Concept Extraction in Breast Ultrasound Reports

TL;DR

This paper introduces BURExtract-Llama, an in-house fine-tuned LLM that extracts clinical information from breast ultrasound reports with performance comparable to GPT-4, offering cost and privacy benefits.

Contribution

The study develops a cost-effective, privacy-preserving LLM for clinical report extraction by fine-tuning Llama3-8B using GPT-4 generated labels, achieving high accuracy.

Findings

Achieved an average F1 score of 84.6% on clinician-annotated reports.

Demonstrated that in-house LLMs can match GPT-4 performance.

Showed potential for cost reduction and improved data privacy.

Abstract

Breast ultrasound is essential for detecting and diagnosing abnormalities, with radiology reports summarizing key findings like lesion characteristics and malignancy assessments. Extracting this critical information is challenging due to the unstructured nature of these reports, with varied linguistic styles and inconsistent formatting. While proprietary LLMs like GPT-4 are effective, they are costly and raise privacy concerns when handling protected health information. This study presents a pipeline for developing an in-house LLM to extract clinical information from radiology reports. We first use GPT-4 to create a small labeled dataset, then fine-tune a Llama3-8B model on it. Evaluated on clinician-annotated reports, our model achieves an average F1 score of 84.6%, which is on par with GPT-4. Our findings demonstrate the feasibility of developing an in-house LLM that not only matches GPT-4's performance but also offers cost reductions and enhanced data privacy.