Assessing Automated Fact-Checking for Medical LLM Responses with Knowledge Graphs

TL;DR

This paper explores using medical knowledge graphs to automatically evaluate the factual accuracy of responses generated by large language models in healthcare, aiming to improve reliability and trustworthiness.

Contribution

The paper introduces FAITH, a novel KG-based framework for factuality evaluation of medical LLM responses without needing reference answers.

Findings

KG-grounded evaluation correlates better with clinician judgments

Effective in distinguishing LLM capabilities

Robust to textual variances

Abstract

The recent proliferation of large language models (LLMs) holds the potential to revolutionize healthcare, with strong capabilities in diverse medical tasks. Yet, deploying LLMs in high-stakes healthcare settings requires rigorous verification and validation to understand any potential harm. This paper investigates the reliability and viability of using medical knowledge graphs (KGs) for the automated factuality evaluation of LLM-generated responses. To ground this investigation, we introduce FAITH, a framework designed to systematically probe the strengths and limitations of this KG-based approach. FAITH operates without reference answers by decomposing responses into atomic claims, linking them to a medical KG, and scoring them based on evidence paths. Experiments on diverse medical tasks with human subjective evaluations demonstrate that KG-grounded evaluation achieves considerably higher correlations with clinician judgments and can effectively distinguish LLMs with varying capabilities. It is also robust to textual variances. The inherent explainability of its scoring can further help users understand and mitigate the limitations of current LLMs. We conclude that while limitations exist, leveraging KGs is a prominent direction for automated factuality assessment in healthcare.