CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale

TL;DR

CheXPO introduces a preference optimization method for chest X-ray vision-language models that leverages counterfactual rationales and similarity-based hard example mining to improve reliability and performance with minimal expert annotation.

Contribution

It proposes a novel CheXPO strategy combining confidence analysis, similarity retrieval, and counterfactual rationales for effective preference optimization in medical VLMs.

Findings

Achieves 8.93% relative performance gain with only 5% of training samples.

Reaches state-of-the-art results across multiple clinical tasks.

Provides a scalable, interpretable approach for radiology applications.

Abstract

Vision-language models (VLMs) are prone to hallucinations that critically compromise reliability in medical applications. While preference optimization can mitigate these hallucinations through clinical feedback, its implementation faces challenges such as clinically irrelevant training samples, imbalanced data distributions, and prohibitive expert annotation costs. To address these challenges, we introduce CheXPO, a Chest X-ray Preference Optimization strategy that combines confidence-similarity joint mining with counterfactual rationale. Our approach begins by synthesizing a unified, fine-grained multi-task chest X-ray visual instruction dataset across different question types for supervised fine-tuning (SFT). We then identify hard examples through token-level confidence analysis of SFT failures and use similarity-based retrieval to expand hard examples for balancing preference sample distributions, while synthetic counterfactual rationales provide fine-grained clinical preferences, eliminating the need for additional expert input. Experiments show that CheXPO achieves 8.93% relative performance gain using only 5% of SFT samples, reaching state-of-the-art performance across diverse clinical tasks and providing a scalable, interpretable solution for real-world radiology applications.