High-Fidelity Pseudo-label Generation by Large Language Models for Training Robust Radiology Report Classifiers

TL;DR

This paper presents DeBERTa-RAD, a two-stage framework that uses large language models for high-quality pseudo-labeling and knowledge distillation to train fast, accurate radiology report classifiers, significantly improving over existing methods.

Contribution

The paper introduces a novel two-stage approach combining LLM pseudo-labeling with DeBERTa-based knowledge distillation for efficient and accurate chest X-ray report classification.

Findings

Achieves a state-of-the-art Macro F1 score of 0.9120 on MIMIC-500.

Outperforms rule-based systems, fine-tuned transformers, and direct LLM inference.

Demonstrates strong handling of uncertain findings in reports.

Abstract

Automated labeling of chest X-ray reports is essential for enabling downstream tasks such as training image-based diagnostic models, population health studies, and clinical decision support. However, the high variability, complexity, and prevalence of negation and uncertainty in these free-text reports pose significant challenges for traditional Natural Language Processing methods. While large language models (LLMs) demonstrate strong text understanding, their direct application for large-scale, efficient labeling is limited by computational cost and speed. This paper introduces DeBERTa-RAD, a novel two-stage framework that combines the power of state-of-the-art LLM pseudo-labeling with efficient DeBERTa-based knowledge distillation for accurate and fast chest X-ray report labeling. We leverage an advanced LLM to generate high-quality pseudo-labels, including certainty statuses, for a large corpus of reports. Subsequently, a DeBERTa-Base model is trained on this pseudo-labeled data using a tailored knowledge distillation strategy. Evaluated on the expert-annotated MIMIC-500 benchmark, DeBERTa-RAD achieves a state-of-the-art Macro F1 score of 0.9120, significantly outperforming established rule-based systems, fine-tuned transformer models, and direct LLM inference, while maintaining a practical inference speed suitable for high-throughput applications. Our analysis shows particular strength in handling uncertain findings. This work demonstrates a promising path to overcome data annotation bottlenecks and achieve high-performance medical text processing through the strategic combination of LLM capabilities and efficient student models trained via distillation.