RadAnnotate: Large Language Models for Efficient and Reliable Radiology Report Annotation

TL;DR

RadAnnotate leverages large language models with retrieval-augmented synthetic reports and confidence-based automation to efficiently and reliably annotate radiology reports, significantly reducing expert effort and improving annotation quality.

Contribution

This work introduces RadAnnotate, a novel LLM-based framework that combines synthetic report generation and confidence thresholds to enhance radiology report annotation efficiency and accuracy.

Findings

Synthetic report models match gold-standard models within 1-2 F1 points.

Synthetic augmentation improves F1 from 0.61 to 0.70 for uncertain observations.

RadAnnotate automatically annotates 55-90% of reports with high confidence.

Abstract

Radiology report annotation is essential for clinical NLP, yet manual labeling is slow and costly. We present RadAnnotate, an LLM-based framework that studies retrieval-augmented synthetic reports and confidence-based selective automation to reduce expert effort for labeling in RadGraph. We study RadGraph-style entity labeling (graph nodes) and leave relation extraction (edges) to future work. First, we train entity-specific classifiers on gold-standard reports and characterize their strengths and failure modes across anatomy and observation categories, with uncertain observations hardest to learn. Second, we generate RAG-guided synthetic reports and show that synthetic-only models remain within 1-2 F1 points of gold-trained models, and that synthetic augmentation is especially helpful for uncertain observations in a low-resource setting, improving F1 from 0.61 to 0.70. Finally, by learning entity-specific confidence thresholds, RadAnnotate can automatically annotate 55-90% of reports at 0.86-0.92 entity match score while routing low-confidence cases for expert review.