CoTBox-TTT: Grounding Medical VQA with Visual Chain-of-Thought Boxes During Test-time Training

TL;DR

CoTBox-TTT enhances medical visual question answering by adaptively grounding evidence during inference through a label-free, prompt-based approach, significantly improving accuracy under domain shifts.

Contribution

It introduces a novel test-time training method that adapts vision-language models using visual chain-of-thought signals without additional labels.

Findings

Increases accuracy by 12.3% on pathVQA.

Effective under domain shift conditions.

Plug-and-play with various backbones.

Abstract

Medical visual question answering could support clinical decision making, yet current systems often fail under domain shift and produce answers that are weakly grounded in image evidence. This reliability gap arises when models attend to spurious regions and when retraining or additional labels are impractical at deployment time. We address this setting with CoTBox-TTT, an evidence-first test-time training approach that adapts a vision-language model at inference while keeping all backbones frozen. The method updates only a small set of continuous soft prompts. It identifies question-relevant regions through a visual chain-of-thought signal and encourages answer consistency across the original image and a localized crop. The procedure is label free, and plug and play with diverse backbones. Experiments on medical VQA show that the approach is practical for real deployments. For instance, adding CoTBox-TTT to LLaVA increases closed-ended accuracy by 12.3% on pathVQA.