MedMosaic: A Challenging Large Scale Benchmark of Diverse Medical Audio

TL;DR

MedMosaic is a comprehensive medical audio question-answering benchmark dataset designed to evaluate reasoning models in realistic clinical scenarios, highlighting current limitations and the need for advanced models.

Contribution

The paper introduces MedMosaic, a large-scale, diverse medical audio dataset with 46,701 QA pairs, to benchmark and analyze reasoning capabilities of multimodal models in medical contexts.

Findings

State-of-the-art models achieve only around 68.1% accuracy.

Reasoning remains challenging across all evaluated systems.

Performance varies significantly across different question types.

Abstract

We present MedMosaic, a medical audio question-answering dataset designed to benchmark language and audio reasoning models under realistic clinical constraints. Medical audio data is difficult to collect due to privacy regulations and high annotation costs arising from domain expertise. Thus, existing benchmarks tend to underrepresent complex medical audio scenarios. To address these challenges, MedMosaic features a diverse range of medical audio types, including condition-related physiological sounds, carefully constructed synthetic voices to mimic speech with artifacts as well as real short and long length clinical conversations to model varying context lengths. The dataset also features a total of 46,701 question-answer pairs, spanning categories such as multiple-choice, sequential multi-turn, and open-ended question-answers, enabling systematic evaluation of multi-hop reasoning and answer generation capabilities. Benchmarking 13 audio and multimodal reasoning models reveals that reasoning remains challenging for all evaluated systems, with substantial performance variation across question types. In particular, even state-of-the-art model like Gemini-2.5-pro can only achieve 68.1% accuracy approximately. These findings underscore persistent limitations in medical reasoning and highlight the need for more robust, domain-specific multimodal reasoning models.