VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

TL;DR

VeriLLMed is a visual analytics system designed to help developers debug medical large language models by integrating biomedical knowledge graphs to identify and analyze recurring diagnostic errors.

Contribution

The paper introduces VeriLLMed, a novel interactive system that leverages external knowledge graphs to facilitate debugging of medical LLMs, addressing interpretability and error pattern identification.

Findings

VeriLLMed enables identification of relation, branch, and missing errors in medical LLMs.

Expert evaluation shows VeriLLMed improves error detection and understanding.

Case studies demonstrate VeriLLMed's effectiveness in clinical error analysis.

Abstract

Large language models (LLMs) show promise in medical diagnosis, but real-world deployment remains challenging due to high-stakes clinical decisions and imperfect reasoning reliability. As a result, careful inspection of model behavior is essential for assessing whether diagnostic reasoning is reliable and clinically grounded. However, debugging medical LLMs remains difficult. First, developers often lack sufficient medical domain expertise to interpret model errors in clinically meaningful terms. Second, models can fail across a large and diverse set of instances involving different input types, tasks, and reasoning steps, making it challenging for developers to prioritize which errors deserve focused inspection. Third, developers struggle to identify recurring error patterns across cases, as existing debugging practices are largely instance-centric and rely on manual inspection of isolated failures. To address these challenges, we present VeriLLMed, a visual analytics system that integrates external biomedical knowledge to audit and debug medical LLM diagnostic reasoning. VeriLLMed transforms model outputs into comparable reasoning paths, constructs knowledge graph-grounded reference paths, and identifies three recurring classes of diagnosis errors: relation errors, branch errors, and missing errors. Case studies and expert evaluation demonstrate that VeriLLMed helps developers identify clinically implausible reasoning and generate actionable insights that can inform the improvement of medical LLMs.