Enhancing Large Language Models for Clinical Decision Support by Incorporating Clinical Practice Guidelines

TL;DR

This paper explores three novel methods for integrating Clinical Practice Guidelines into Large Language Models to improve clinical decision support, demonstrating enhanced accuracy in COVID-19 outpatient treatment recommendations through both automatic and human evaluations.

Contribution

The study introduces three distinct techniques—BDT, PAGC, and CoT-FSP—for incorporating CPGs into LLMs, and evaluates their effectiveness in a COVID-19 case study.

Findings

Enhanced LLMs outperform baseline zero-shot prompting in clinical tasks.

BDT method yields the best automatic evaluation results.

All proposed methods perform well in human assessments.

Abstract

Background Large Language Models (LLMs), enhanced with Clinical Practice Guidelines (CPGs), can significantly improve Clinical Decision Support (CDS). However, methods for incorporating CPGs into LLMs are not well studied. Methods We develop three distinct methods for incorporating CPGs into LLMs: Binary Decision Tree (BDT), Program-Aided Graph Construction (PAGC), and Chain-of-Thought-Few-Shot Prompting (CoT-FSP). To evaluate the effectiveness of the proposed methods, we create a set of synthetic patient descriptions and conduct both automatic and human evaluation of the responses generated by four LLMs: GPT-4, GPT-3.5 Turbo, LLaMA, and PaLM 2. Zero-Shot Prompting (ZSP) was used as the baseline method. We focus on CDS for COVID-19 outpatient treatment as the case study. Results All four LLMs exhibit improved performance when enhanced with CPGs compared to the baseline ZSP. BDT outperformed both CoT-FSP and PAGC in automatic evaluation. All of the proposed methods demonstrated high performance in human evaluation. Conclusion LLMs enhanced with CPGs demonstrate superior performance, as compared to plain LLMs with ZSP, in providing accurate recommendations for COVID-19 outpatient treatment, which also highlights the potential for broader applications beyond the case study.