Comprehensive Modeling and Question Answering of Cancer Clinical Practice Guidelines using LLMs

TL;DR

This paper presents a method to digitally represent cancer clinical practice guidelines as graphs using automated extraction and LLM-based classification, enabling accurate, constrained question answering in the medical domain.

Contribution

It introduces a novel approach combining graph modeling and LLMs for faithful, semantic enrichment of cancer guidelines and constrained question answering.

Findings

Achieved node classification accuracy of 80.86% with zero-shot learning.

Achieved node classification accuracy of 88.47% with few-shot learning.

Developed a method for factual, constrained question answering using subgraph extraction.

Abstract

The updated recommendations on diagnostic procedures and treatment pathways for a medical condition are documented as graphical flows in Clinical Practice Guidelines (CPGs). For effective use of the CPGs in helping medical professionals in the treatment decision process, it is necessary to fully capture the guideline knowledge, particularly the contexts and their relationships in the graph. While several existing works have utilized these guidelines to create rule bases for Clinical Decision Support Systems, limited work has been done toward directly capturing the full medical knowledge contained in CPGs. This work proposes an approach to create a contextually enriched, faithful digital representation of National Comprehensive Cancer Network (NCCN) Cancer CPGs in the form of graphs using automated extraction and node & relationship classification. We also implement semantic enrichment of the model by using Large Language Models (LLMs) for node classification, achieving an accuracy of 80.86% and 88.47% with zero-shot learning and few-shot learning, respectively. Additionally, we introduce a methodology for answering natural language questions with constraints to guideline text by leveraging LLMs to extract the relevant subgraph from the guideline knowledge base. By generating natural language answers based on subgraph paths and semantic information, we mitigate the risk of incorrect answers and hallucination associated with LLMs, ensuring factual accuracy in medical domain Question Answering.