Enhancing Clinical Multiple-Choice Questions Benchmarks with Knowledge Graph Guided Distractor Generation

TL;DR

This paper introduces a knowledge graph-guided method to generate more challenging distractors for clinical MCQ benchmarks, improving the evaluation of medical language models by making questions more deceptive.

Contribution

We propose a novel knowledge graph-based framework for generating clinically plausible yet misleading distractors to enhance medical MCQ benchmarks.

Findings

KGGDG reduces LLM accuracy on medical benchmarks

Generated distractors are clinically relevant and misleading

Improves robustness of medical LLM evaluations

Abstract

Clinical tasks such as diagnosis and treatment require strong decision-making abilities, highlighting the importance of rigorous evaluation benchmarks to assess the reliability of large language models (LLMs). In this work, we introduce a knowledge-guided data augmentation framework that enhances the difficulty of clinical multiple-choice question (MCQ) datasets by generating distractors (i.e., incorrect choices that are similar to the correct one and may confuse existing LLMs). Using our KG-based pipeline, the generated choices are both clinically plausible and deliberately misleading. Our approach involves multi-step, semantically informed walks on a medical knowledge graph to identify distractor paths-associations that are medically relevant but factually incorrect-which then guide the LLM in crafting more deceptive distractors. We apply the designed knowledge graph guided distractor generation (KGGDG) pipline, to six widely used medical QA benchmarks and show that it consistently reduces the accuracy of state-of-the-art LLMs. These findings establish KGGDG as a powerful tool for enabling more robust and diagnostic evaluations of medical LLMs.