U2-BENCH: Benchmarking Large Vision-Language Models on Ultrasound Understanding

TL;DR

U2-BENCH is a comprehensive benchmark designed to evaluate large vision-language models on ultrasound understanding, covering diverse tasks and scenarios to identify strengths and challenges in medical ultrasound interpretation.

Contribution

This work introduces U2-BENCH, the first extensive benchmark for assessing LVLMs on ultrasound tasks, including 7,241 cases across 15 anatomical regions and 8 clinical tasks.

Findings

Strong performance in image classification

Challenges in spatial reasoning and language generation

Benchmark facilitates targeted improvements in LVLMs for ultrasound

Abstract

Ultrasound is a widely-used imaging modality critical to global healthcare, yet its interpretation remains challenging due to its varying image quality on operators, noises, and anatomical structures. Although large vision-language models (LVLMs) have demonstrated impressive multimodal capabilities across natural and medical domains, their performance on ultrasound remains largely unexplored. We introduce U2-BENCH, the first comprehensive benchmark to evaluate LVLMs on ultrasound understanding across classification, detection, regression, and text generation tasks. U2-BENCH aggregates 7,241 cases spanning 15 anatomical regions and defines 8 clinically inspired tasks, such as diagnosis, view recognition, lesion localization, clinical value estimation, and report generation, across 50 ultrasound application scenarios. We evaluate 23 state-of-the-art LVLMs, both open- and closed-source, general-purpose and medical-specific. Our results reveal strong performance on image-level classification, but persistent challenges in spatial reasoning and clinical language generation. U2-BENCH establishes a rigorous and unified testbed to assess and accelerate LVLM research in the uniquely multimodal domain of medical ultrasound imaging.