CPGPrompt: Translating Clinical Guidelines into LLM-Executable Decision Support

TL;DR

This paper introduces CPGPrompt, an auto-prompting system that converts narrative clinical guidelines into structured decision trees and uses large language models to evaluate patient cases, improving AI integration in healthcare.

Contribution

It presents a novel framework for translating clinical guidelines into LLM-executable decision support, addressing interpretability and domain applicability issues of previous rule-based systems.

Findings

High accuracy in binary referral decisions (F1: 0.85-1.00)

Reduced performance in multi-class pathway classification

Domain-specific challenges like negation and temporal reasoning

Abstract

Clinical practice guidelines (CPGs) provide evidence-based recommendations for patient care; however, integrating them into Artificial Intelligence (AI) remains challenging. Previous approaches, such as rule-based systems, face significant limitations, including poor interpretability, inconsistent adherence to guidelines, and narrow domain applicability. To address this, we develop and validate CPGPrompt, an auto-prompting system that converts narrative clinical guidelines into large language models (LLMs). Our framework translates CPGs into structured decision trees and utilizes an LLM to dynamically navigate them for patient case evaluation. Synthetic vignettes were generated across three domains (headache, lower back pain, and prostate cancer) and distributed into four categories to test different decision scenarios. System performance was assessed on both binary specialty-referral decisions and fine-grained pathway-classification tasks. The binary specialty referral classification achieved consistently strong performance across all domains (F1: 0.85-1.00), with high recall (1.00 $\pm$ 0.00). In contrast, multi-class pathway assignment showed reduced performance, with domain-specific variations: headache (F1: 0.47), lower back pain (F1: 0.72), and prostate cancer (F1: 0.77). Domain-specific performance differences reflected the structure of each guideline. The headache guideline highlighted challenges with negation handling. The lower back pain guideline required temporal reasoning. In contrast, prostate cancer pathways benefited from quantifiable laboratory tests, resulting in more reliable decision-making.