Clinical Knowledge Graph Construction and Evaluation with Multi-LLMs via Retrieval-Augmented Generation

TL;DR

This paper presents a novel framework that leverages multi-LLMs and retrieval-augmented generation to construct and evaluate clinical knowledge graphs directly from unstructured text, improving accuracy and semantic consistency in oncology data.

Contribution

The authors introduce an end-to-end, schema-constrained, multi-LLM-based approach for clinical KG construction that includes validation and iterative refinement, addressing limitations of prior methods.

Findings

Achieved higher precision and relevance in clinical KGs.

Produced ontology-aligned, SPARQL-compatible graphs without gold-standard annotations.

Demonstrated effectiveness in oncology cohorts for improved knowledge graph quality.

Abstract

Large language models (LLMs) offer new opportunities for constructing knowledge graphs (KGs) from unstructured clinical narratives. However, existing approaches often rely on structured inputs and lack robust validation of factual accuracy and semantic consistency, limitations that are especially problematic in oncology. We introduce an end-to-end framework for clinical KG construction and evaluation directly from free text using multi-agent prompting and a schema-constrained Retrieval-Augmented Generation (KG-RAG) strategy. Our pipeline integrates (1) prompt-driven entity, attribute, and relation extraction; (2) entropy-based uncertainty scoring; (3) ontology-aligned RDF/OWL schema generation; and (4) multi-LLM consensus validation for hallucination detection and semantic refinement. Beyond static graph construction, the framework supports continuous refinement and self-supervised evaluation, enabling iterative improvement of graph quality. Applied to two oncology cohorts (PDAC and BRCA), our method produces interpretable, SPARQL-compatible, and clinically grounded knowledge graphs without relying on gold-standard annotations. Experimental results demonstrate consistent gains in precision, relevance, and ontology compliance over baseline methods.