MedEyes: Learning Dynamic Visual Focus for Medical Progressive Diagnosis

TL;DR

MedEyes introduces a reinforcement learning framework that mimics clinician visual reasoning by dynamically attending to medical images, integrating expert guidance and dual-mode exploration to improve medical visual question answering accuracy.

Contribution

It presents MedEyes, a novel RL-based model with a gaze-guided reasoning navigator and confidence sampler, enhancing clinical reasoning and interpretability in medical AI.

Findings

Achieved +8.5 percentage points improvement on medical VQA benchmarks.

Effectively models clinician-like visual search and reasoning processes.

Demonstrates robustness and trustworthiness in medical diagnosis tasks.

Abstract

Accurate medical diagnosis often involves progressive visual focusing and iterative reasoning, characteristics commonly observed in clinical workflows. While recent vision-language models demonstrate promising chain-of-thought (CoT) reasoning capabilities via reinforcement learning with verifiable rewards (RLVR), their purely on-policy learning paradigm tends to reinforce superficially coherent but clinically inaccurate reasoning paths. We propose MedEyes, a novel reinforcement learning framework that dynamically models clinician-style diagnostic reasoning by progressively attending to and interpreting relevant medical image regions. By incorporating off-policy expert guidance, MedEyes converts expert visual search trajectories into structured external behavioral signals, guiding the model toward clinically aligned visual reasoning. We design the Gaze-guided Reasoning Navigator (GRN) to emulate the diagnostic process through a dual-mode exploration strategy, scanning for systematic abnormality localization and drilling for detailed regional analysis. To balance expert imitation and autonomous discovery, we introduce the Confidence Value Sampler (CVS), which employs nucleus sampling and adaptive termination to create diverse yet credible exploration paths. Finally, the dual-stream GRPO optimization framework decouples on-policy and off-policy learning signals, mitigating reward assimilation and entropy collapse. Experiments demonstrate that MedEyes achieves an average performance improvement of +8.5pp across multiple medical VQA benchmarks, validating MedEyes's potential in building trustworthy medical AI systems. Code is available at https://github.com/zhcz328/MedEyes.