S-Chain: Structured Visual Chain-of-Thought For Medicine

TL;DR

S-Chain introduces a large-scale, expert-annotated dataset with structured visual reasoning for medical VLMs, significantly enhancing interpretability, grounding, and robustness in medical visual question answering.

Contribution

It provides the first large-scale dataset with structured visual CoT annotations in medical images, enabling improved reasoning and grounding in medical vision-language models.

Findings

SV-CoT supervision improves interpretability and grounding fidelity.

Benchmarking shows state-of-the-art models benefit from SV-CoT training.

Proposed mechanisms enhance alignment between visual evidence and reasoning.

Abstract

Faithful reasoning in medical vision-language models (VLMs) requires not only accurate predictions but also transparent alignment between textual rationales and visual evidence. While Chain-of-Thought (CoT) prompting has shown promise in medical visual question answering (VQA), no large-scale expert-level dataset has captured stepwise reasoning with precise visual grounding. We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT), explicitly linking visual regions to reasoning steps. The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability. Using S-Chain, we benchmark state-of-the-art medical VLMs (ExGra-Med, LLaVA-Med) and general-purpose VLMs (Qwen2.5-VL, InternVL2.5), showing that SV-CoT supervision significantly improves interpretability, grounding fidelity, and robustness. Beyond benchmarking, we study its synergy with retrieval-augmented generation, revealing how domain knowledge and visual grounding interact during autoregressive reasoning. Finally, we propose a new mechanism that strengthens the alignment between visual evidence and reasoning, improving both reliability and efficiency. S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical VLMs.