CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency

TL;DR

CheXPO-v2 introduces a knowledge graph-based reward system for aligning medical vision-language models, significantly improving factual accuracy and reasoning transparency in chest X-ray analysis with high data efficiency.

Contribution

It presents a novel process supervision framework using entity-relation matching, enhancing model reliability and interpretability over outcome-based methods.

Findings

Outperforms GRPO and state-of-the-art models on MIMIC-CXR-VQA

Achieves new state-of-the-art accuracy with only 5k samples

Produces clinically sound, verifiable reasoning

Abstract

Medical Vision-Language Models (VLMs) are prone to hallucinations, compromising clinical reliability. While reinforcement learning methods like Group Relative Policy Optimization (GRPO) offer a low-cost alignment solution, their reliance on sparse, outcome-based rewards inadvertently encourages models to "overthink" -- generating verbose, convoluted, and unverifiable Chain-of-Thought reasoning to justify answers. This focus on outcomes obscures factual errors and poses significant safety risks. To address this, we propose CheXPO-v2, a novel alignment framework that shifts from outcome to process supervision. Our core innovation is a Knowledge Graph Consistency Reward mechanism driven by Entity-Relation Matching. By explicitly parsing reasoning steps into structured "Disease, Relation, Anatomy" triplets, we provide fine-grained supervision that penalizes incoherent logic and hallucinations at the atomic level. Integrating this with a hard-example mining strategy, our approach significantly outperforms GRPO and state-of-the-art models on benchmarks like MIMIC-CXR-VQA. Crucially, CheXPO-v2 achieves new state-of-the-art accuracy using only 5k samples, demonstrating exceptional data efficiency while producing clinically sound and verifiable reasoning. The project source code is publicly available at: https://github.com/ecoxial2007/CheX-Phi4MM.