Can Rule-Based Insights Enhance LLMs for Radiology Report Classification? Introducing the RadPrompt Methodology

TL;DR

This paper introduces RadPrompt, a novel approach combining rule-based systems and large language models to improve radiology report classification, demonstrating significant performance gains on clinical datasets.

Contribution

The paper presents RadPert, an enhanced rule-based label extraction system, and RadPrompt, a multi-turn prompting strategy that significantly boosts LLMs' zero-shot classification performance.

Findings

RadPrompt outperforms baseline models including GPT-4 Turbo.

RadPert improves label extraction robustness against syntactic variability.

The combined approach achieves higher F1 scores on clinical datasets.

Abstract

Developing imaging models capable of detecting pathologies from chest X-rays can be cost and time-prohibitive for large datasets as it requires supervision to attain state-of-the-art performance. Instead, labels extracted from radiology reports may serve as distant supervision since these are routinely generated as part of clinical practice. Despite their widespread use, current rule-based methods for label extraction rely on extensive rule sets that are limited in their robustness to syntactic variability. To alleviate these limitations, we introduce RadPert, a rule-based system that integrates an uncertainty-aware information schema with a streamlined set of rules, enhancing performance. Additionally, we have developed RadPrompt, a multi-turn prompting strategy that leverages RadPert to bolster the zero-shot predictive capabilities of large language models, achieving a statistically significant improvement in weighted average F1 score over GPT-4 Turbo. Most notably, RadPrompt surpasses both its underlying models, showcasing the synergistic potential of LLMs with rule-based models. We have evaluated our methods on two English Corpora: the MIMIC-CXR gold-standard test set and a gold-standard dataset collected from the Cambridge University Hospitals.