Medical Image Spatial Grounding with Semantic Sampling

TL;DR

This paper introduces MIS-Ground, a comprehensive benchmark for evaluating medical image spatial grounding in vision language models, and proposes MIS-SemSam, a semantic sampling method that enhances grounding accuracy.

Contribution

The study presents a new benchmark for assessing VLMs in medical spatial grounding and introduces a model-agnostic optimization technique that significantly improves performance.

Findings

MIS-SemSam improves Qwen3-VL-32B accuracy by 13.06%.

Varying visual and textual prompts affect spatial grounding performance.

MIS-Ground enables reproducible evaluation of VLM vulnerabilities.

Abstract

Vision language models (VLMs) have shown significant promise in visual grounding for images as well as videos. In medical imaging research, VLMs represent a bridge between object detection and segmentation, and report understanding and generation. However, spatial grounding of anatomical structures in the three-dimensional space of medical images poses many unique challenges. In this study, we examine image modalities, slice directions, and coordinate systems as differentiating factors for vision components of VLMs, and the use of anatomical, directional, and relational terminology as factors for the language components. We then demonstrate that visual and textual prompting systems such as labels, bounding boxes, and mask overlays have varying effects on the spatial grounding ability of VLMs. To enable measurement and reproducibility, we introduce MIS-Ground, a benchmark that comprehensively tests a VLM for vulnerabilities against specific modes of Medical Image Spatial Grounding. We release MIS-Ground to the public at https://anonymous.4open.science/r/mis-ground. In addition, we present MIS-SemSam, a low-cost, inference-time, and model-agnostic optimization of VLMs that improve their spatial grounding ability with the use of Semantic Sampling. We find that MIS-SemSam improves the accuracy of Qwen3-VL-32B on MIS-Ground by 13.06%.