Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zooming

TL;DR

This paper introduces a confidence-aware ultrasound VQA framework that mimics sonographers' focus on lesions through active zooming and uncertainty estimation, improving lesion localization accuracy.

Contribution

It proposes a structured Zoom-then-Diagnose paradigm combined with an uncertainty-aware reward within GRPO to enhance model confidence and interpretability in ultrasound diagnosis.

Findings

Improved lesion localization by 39.3% across multiple datasets.

Model learns to actively focus on lesions before diagnosis.

Incorporates uncertainty estimation to handle ambiguous cases.

Abstract

Vision-Language Models (VLMs) have significantly advanced medical visual question answering, yet their performance in ultrasound remains suboptimal. In clinical practice, sonographers explicitly focus on lesion regions to formulate reports, though diagnostic interpretations sometimes vary due to inherent subjectivity. However, existing VLMs are not explicitly structured to interactively zoom into lesions prior to diagnosis; moreover, they typically treat annotations as unbiased ground truths, failing to account for their inherent subjectivity and ambiguity. In this paper, we propose a framework specifically designed to consider the sonographer's cognitive workflow. We first introduce a structured Zoom-then-Diagnose paradigm, which replicates the interactive search process to enable lesion-focused reasoning. Furthermore, within the Group Relative Policy Optimization (GRPO) framework, we introduce an uncertainty-aware reward derived from stochastic group-wise rollouts to estimate prediction consistency as a proxy for model confidence. Together, these two components encourage the model to reinforce accurate predictions on clear cases while remaining cautious under ambiguity. Experiments across liver, breast, and thyroid datasets show that our framework improves lesion localization by 39.3\%, demonstrating that our model has learned the ability to actively look closer and diagnose.