PatientVLM Meets DocVLM: Pre-Consultation Dialogue Between Vision-Language Models for Efficient Diagnosis

TL;DR

This paper introduces a novel framework where vision-language models simulate doctor-patient dialogues to improve medical diagnosis accuracy by incorporating symptom information, validated through clinical feedback.

Contribution

It proposes a pre-consultation dialogue framework with two VLMs, enabling realistic symptom elicitation and enhancing diagnostic performance beyond image-only methods.

Findings

Dialogue-based supervision improves diagnosis accuracy

Clinicians confirm the realism of synthetic symptoms

Coherent multi-turn diagnostic interactions are achieved

Abstract

Traditionally, AI research in medical diagnosis has largely centered on image analysis. While this has led to notable advancements, the absence of patient-reported symptoms continues to hinder diagnostic accuracy. To address this, we propose a Pre-Consultation Dialogue Framework (PCDF) that mimics real-world diagnostic procedures, where doctors iteratively query patients before reaching a conclusion. Specifically, we simulate diagnostic dialogues between two vision-language models (VLMs): a DocVLM, which generates follow-up questions based on the image and dialogue history, and a PatientVLM, which responds using a symptom profile derived from the ground-truth diagnosis. We additionally conducted a small-scale clinical validation of the synthetic symptoms generated by our framework, with licensed clinicians confirming their clinical relevance, symptom coverage, and overall realism. These findings indicate that the resulting DocVLM-PatientVLM interactions form coherent, multi-turn consultations paired with images and diagnoses, which we then use to fine-tune the DocVLM. This dialogue-based supervision leads to substantial gains over image-only training, highlighting the value of realistic symptom elicitation for diagnosis.