See-in-Pairs: Reference Image-Guided Comparative Vision-Language Models for Medical Diagnosis

TL;DR

This paper demonstrates that incorporating healthy reference images and comparative prompts into vision-language models significantly improves medical diagnosis accuracy, with benefits from lightweight fine-tuning and various reference selection strategies.

Contribution

It introduces a reference image-guided approach for medical VLMs, enhancing diagnostic performance through comparative analysis and practical reference selection methods.

Findings

Improved diagnostic accuracy with reference images and prompts

Lightweight fine-tuning further boosts performance

Consistent results across different reference selection strategies

Abstract

Medical image diagnosis is challenging because many diseases resemble normal anatomy and exhibit substantial interpatient variability. Clinicians routinely rely on comparative diagnosis, such as referencing cross-patient healthy control images to identify subtle but clinically meaningful abnormalities. Although healthy reference images are abundant in practice, existing medical vision-language models (VLMs) primarily operate in a single-image or single-series setting and lack explicit mechanisms for comparative diagnosis. This work investigates whether incorporating clinically motivated comparison can enhance VLM performance. We show that providing VLMs with both a query image and a matched healthy reference image, accompanied by cross-patient comparative prompts, significantly improves diagnostic performance. This performance can be further augmented by lightweight supervised fine-tuning (SFT) on a small amount of data. At the same time, we evaluate multiple strategies for selecting reference images, including random sampling, demographic attribute matching, embedding-based retrieval, and cross-center selection, and find consistently strong performance across all settings. Finally, we investigate why comparative diagnosis is effective theoretically, and observe improved sample efficiency and tighter alignment between visual and textual representations. Our findings highlight the clinical relevance of comparison-based diagnosis, provide practical strategies for incorporating reference images into VLMs, and demonstrate improved performance across diverse medical imaging tasks.