Towards Spoken Mathematical Reasoning: Benchmarking Speech-based Models over Multi-faceted Math Problems

TL;DR

This paper introduces Spoken-MQA, a benchmark for evaluating speech-based models' mathematical reasoning, revealing current models' strengths in contextual reasoning but weaknesses in direct arithmetic and verbalized expressions.

Contribution

It presents the first comprehensive benchmark for speech-based mathematical reasoning, assessing both cascade and end-to-end speech LLMs across diverse math problems.

Findings

Speech LLMs perform well on contextual reasoning with basic arithmetic.

Models struggle with direct arithmetic problems.

Current speech LLMs have difficulty interpreting verbalized math expressions.

Abstract

Recent advances in large language models (LLMs) and multimodal LLMs (MLLMs) have led to strong reasoning ability across a wide range of tasks. However, their ability to perform mathematical reasoning from spoken input remains underexplored. Prior studies on speech modality have mostly focused on factual speech understanding or simple audio reasoning tasks, providing limited insight into logical step-by-step reasoning, such as that required for mathematical problem solving. To address this gap, we introduce Spoken Math Question Answering (Spoken-MQA), a new benchmark designed to evaluate the mathematical reasoning capabilities of speech-based models, including both cascade models (ASR + LLMs) and end-to-end speech LLMs. Spoken-MQA covers a diverse set of math problems, including pure arithmetic, single-step and multi-step contextual reasoning, and knowledge-oriented reasoning problems, all presented in unambiguous natural spoken language. Through extensive experiments, we find that: (1) while some speech LLMs perform competitively on contextual reasoning tasks involving basic arithmetic, they still struggle with direct arithmetic problems; (2) current LLMs exhibit a strong bias toward symbolic mathematical expressions written in LaTex and have difficulty interpreting verbalized mathematical expressions; and (3) mathematical knowledge reasoning abilities are significantly degraded in current speech LLMs.