AHELM: A Holistic Evaluation of Audio-Language Models

TL;DR

AHELM is a comprehensive benchmark for audio-language models that evaluates multiple capabilities and aspects such as fairness, safety, and reasoning, using standardized prompts and metrics across diverse datasets.

Contribution

This paper introduces AHELM, a holistic benchmark with new synthetic datasets, standardized evaluation protocols, and a broad assessment of audio-language models' performance.

Findings

Gemini 2.5 Pro ranks top in 5 aspects but shows group unfairness.

Baseline systems perform reasonably well, with one ranking 6th overall.

AHELM provides a standardized, multi-aspect evaluation framework.

Abstract

Evaluations of audio-language models (ALMs) -- multimodal models that take interleaved audio and text as input and output text -- are hindered by the lack of standardized benchmarks; most benchmarks measure only one or two capabilities and omit evaluative aspects such as fairness or safety. Furthermore, comparison across models is difficult as separate evaluations test a limited number of models and use different prompting methods and inference parameters. To address these shortfalls, we introduce AHELM, a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE, which evaluates the ALMs on avoiding stereotypes, and CoRe-Bench, which measures reasoning over conversational audio through inferential multi-turn question answering -- to holistically measure the performance of ALMs across 10 aspects we have identified as important to the development and usage of ALMs: audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, and safety. We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models. We test 14 open-weight and closed-API ALMs from 3 developers and 3 additional simple baseline systems each consisting of an automatic speech recognizer and a language model. Our results show that while Gemini 2.5 Pro ranks top in 5 out of 10 aspects, it exhibits group unfairness ($p=0.01$) on ASR tasks whereas most of the other models do not. We also find that the baseline systems perform reasonably well on AHELM, with one ranking 6th overall despite having only speech-to-text capabilities. For transparency, all raw prompts, model generations, and outputs are available on our website at https://crfm.stanford.edu/helm/audio/v1.0.0. AHELM is intended to be a living benchmark and new datasets and models will be added over time.

Tables

Table 1. Table 1 : Evaluative aspects in AHELM. See Section ˜ 3.1 and Figure ˜ 1 for details and examples.

Aspect	Description
Audio Perception	Extracting meaningful information from audio signals
Knowledge	Recalling facts or information contained in the ALM
Reasoning	Performing a series of logical inferences to deduce an answer
Emotion detection	Detecting the user’s conscious mental state deriving from his mood, circumstances, or relationships with others
Bias	Prevent forming inappropriate or unwarranted associations between the input and output of the model
Fairness	Ensuring that the model’s responses remain consistent when a non-essential or spurious attribute (e.g., sex) of the input is altered (i.e., counterfactual fairness) or having uniform performance on every subset of the data when an attribute is used as the filter (i.e., performance disparity)
Multilinguality	Executing tasks effectively even when the language of the instructions or the language of the output is altered
Robustness	Generating accurate and desired outputs despite variations or disturbances in the input audio (e.g., noise) and/or text (e.g., typos)
Toxicity	Detecting and steering clear of offensive or harmful content (e.g., hate speech, violent language, abusive remarks)
Safety	Refusing to generate responses that could potentially harm humans

Table 2. Table 2 : List of scenarios used in AHELM. * indicates adaptation to test for fairness. ** indicates new scenario introduced in this paper.

Aspect	Scenarios	Category	Description	Metrics
Auditory perception	AudioCaps[25]		AudioCaps contains 46K audio clips to human-written text pairs. The audio clips are from AudioSet and covers a wide range of human and animal sounds, musical instruments and genres, and common everyday environmental sounds. The captions are collected via crowdsourcing. This scenario measures how well the ALM can express sounds in various settings as text.	GPT-4o judge critique
	VoxCeleb2 [7]	Audio	VoxCeleb2 contains over 1M utterances by celebrities collected from YouTube. We use only the audio subset. This scenario measures whether the ALM can decipher whether the speakers in two audio clips are the same.	Exact match
	VocalSound [19]		VocalSound consists of >21,000 crowdsourced recordings of laughter, sighs, coughs, throat clearing, sneezes, and sniffs from 3,365 unique subjects. It tests whether the ALMs can recognize the aforementioned human sounds.	Exact match
	LibriSpeech [33]		The LibriSpeech corpus is derived from audiobooks that are part of the LibriVox project. This corpus is one of the most widely-used ASR corpus, which has been extended to many applications such as robust ASR and multilingual ASR tasks. The dataset contains the audio and transcriptions and assesses automated speech recognition capabilities.	WER
Knowledge	AIR-Bench[44] (Foundation)	Music Genre Recognition, Music Instrument Classification, Music QA	AIR-Bench (Foundation) which consists of 19 tasks with approximately 19k single-choice questions. We use only the music-related subsets to test music understanding.	Exact match
	AIR-Bench[44] (Chat)	Music, Sound	AIR-Bench (Chat) contains 2k instances of open-ended question-and-answer data. This benchmark evaluates the ability of audio language models to understand various types of audio signals (including human speech, natural sounds and music) and to interact with humans through text.	GPT-4o judge critique
Reasoning	AIR-Bench[44] (Chat)	Mixed, Speech	These subsets of AIR-Bench test the ability of models to reason with speech and sounds.	GPT-4o judge critique
	CoRe-Bench**		CoRe-Bench contains a diverse range of audio conversations and questions whose answers can be inferred from the conversations.	Pseudo-exact match
Emotion detection	MELD [34]	Audio	Multimodal EmotionLines Dataset (MELD) is created by enhancing and extending EmotionLines dataset. MELD has more than 1,400 dialogues and 13,000 utterances from Friends TV series. Multiple speakers participated in the dialogues. Each utterance in a dialogue has been labeled by any of these seven emotions - Anger, Disgust, Sadness, Joy, Neutral, Surprise and Fear. The task is to classify the emotion after listening to an audio clip.	Exact match
	MUStARD [5]		MUStARD is a multimodal video corpus focusing on automated sarcasm discovery. It consists of audiovisual utterances from sitcoms such as Friends, The Golden Girls, The Big Bang Theory, and Sarcasmaholics Anonymous. Sarcasm labels are labeled by humans. Each utterance is accompanied by a context that provides additional information on the scenario where it occurs. We use only the audio from the videos to evaluate how well ALMs detect sarcasm in speech.	Exact match
Bias	PARADE**	{Status, Occupation} $\times$ {Male, Female}	PARADE is a new audio-text multiple-choice QA benchmark consisting of 436 instances that explores occupational and status bias in ALMs.	Exact match
Fairness	FLEURS[9] (ASR)*	Female vs Male	FLEURS is an $n$-way parallel speech dataset in 102 languages built on top of the machine translation FLoRes-101 benchmark. We evaluate the mean WER between male and female speakers in order to test the difference in the models’ ASR abilities when confronted with speech from different sexes.	WER
	LibriSpeech* [33]	Female vs Male	Similar to the previously mentioned LibriSpeech, except that we ask the model to do ASR on audio files from different sexes. This scenario measures how the ASR capability of ALMs is affected by different sexes.	WER
Multilinguality	CoVoST 2[41]	Spanish$\to$English, Chinese$\to$English	CoVost-2 is a large-scale multilingual speech translation corpus covering translations from 21 languages into English and from English into various languages. We use the Spanish-to-English and Chinese-to-English subsets to test for the ability to translate speech from a language to a target language.	BLEU
	FLEURS[9]	Finnish, Mandarin_chinese, Thai, Hebrew, Bengali, English, Zulu	We use the audio and transcriptions to test for the ability to transcribe audio in various languages.	WER
	Multilingual LibriSpeech [35]	Italian, French, Polish, Dutch, Portuguese, Spanish, German	The Multilingual LibriSpeech dataset is derived from audiobooks in LibriVox and consists of $\sim$ 44.5K hours of English and a total of $\sim$6K hours for other 7 languages. The task is to transcribe audio in various languages.	WER
Robustness	Speech Robust Bench (LibriSpeech-Clean) [37]	{Gaussian Noise, Environment Noise} $\times$ {Levels 1, 2, 3}	Speech Robust Bench (SRB) comprises of 114 input perturbations that simulate a heterogeneous range of corruptions that ASR models may encounter when deployed in the wild. In this scenario, we select four subsets in the benchmark for evaluation, each corresponds to a clean version of audio task, to evaluate how well the ALMs can process speech in noisy environments.	WER
Toxicity	MuToX [10]	Estonian, French, Urdu, English, Bulgarian, German, Mandarin Chinese, Indonesian, Turkish, Slovak, Bengali, Arabic, Hindi, Polish, Tagalog, Italian, Catalan, Czech, Hungarian, Greek, Swahili, Danish, Finnish, Hebrew, Russian, Vietnamese, Dutch, Portuguese, Spanish	MuTox consists of $\sim$20k audio utterances for English and Spanish and $\sim$4k for the other languages. This scenario evaluates ALM for zero-shot toxicity detection across a broad range of languages.	Exact match
Safety	Voice jailbreak attacks[38]	Text jailbreak, Baseline	Voice Jailbreak Attacks Against GPT-4o. This scenario test how ALM can resist jailbreak attacks.	Toxic fraction

Table 3. Table 3 : Audio language models evaluated in AHELM. The second block lists models that are used to construct our baseline systems and are not ALMs. A question mark indicates unknown.

Model	Identifier	Creator	Access	Release Date	Parameters	Ref.	Knowledge Cutoff
Gemini 1.5 Pro (001)	gemini-1.5-pro-001	Google	API	2024-05-24	?	[17]	?
Gemini 1.5 Flash (001)	gemini-1.5-flash-001	Google	API	2024-05-24	?	[17]	?
Gemini 1.5 Pro (002)	gemini-1.5-pro-002	Google	API	2024-09-24	?	[17]	?
Gemini 1.5 Flash (002)	gemini-1.5-flash-002	Google	API	2024-09-24	?	[17]	?
Gemini 2.0 Flash (Experimental)	gemini-2.0-flash-exp	Google	API	2024-12-11	?	[30]	?
Gemini 2.0 Flash	gemini-2.0-flash-001	Google	API	2025-02-01	?	[30]	?
Gemini 2.0 Flash Lite	gemini-2.0-flash-lite-001	Google	API	2025-03-25	?	[30]	?
Gemini 2.5 Pro (05-06 preview)	gemini-2.5-pro-preview-05-06	Google	API	2025-05-06	?	[24]	?
Gemini 2.5 Flash (05-20 preview)	gemini-2.5-flash-preview-05-20	Google	API	2025-04-17	?	[24]	?
GPT-4o Audio (Preview 2024-10-01)	gpt-4o-audio-preview-2024-10-01	OpenAI	API	2024-10-01	?	[32]	2023-09-30
GPT-4o Audio (Preview 2024-12-17)	gpt-4o-audio-preview-2024-12-17	OpenAI	API	2024-12-17	?	[32]	2023-09-30
GPT-4o mini Audio (Preview 2024-12-17)	gpt-4o-mini-audio-preview-2024-12-17	OpenAI	API	2024-12-17	?	[32]	2023-09-30
Qwen2-Audio Instruct (7B)	qwen2-audio-7b-instruct	Alibaba Cloud	Open-weight	2024-11-28	8.4B	[6]	?
Qwen2.5-Omni (7B)	qwen2.5-omni-7b	Alibaba Cloud	Open-weight	2025-03-27	10.7B	[42]	?
Whisper 1	whisper-1	OpenAI	API	2022-09-21	?	[36]	?
GPT-4o Transcribe	gpt-4o-transcribe	OpenAI	API	2025-03-20	?	[32]	2024-05-31
GPT-4o Mini Transcribe	gpt-4o-mini-transcribe	OpenAI	API	2025-03-20	?	[32]	2024-05-31
GPT-4o (2024-11-20)	gpt-4o-2024-11-20	OpenAI	API	2024-11-20	?	[31]	2023-09-30

Table 4. Table A1 : Models and aspects evaluated prior to AHELM, compiled to the best of our ability. A tick in the table indicates that the model is tested on the aspect in either one of the benchmark papers, its official technical report, or its blog post at launch. In comparison, AHELM checks every box in the table (indicated by the green background ) and thus, allows holistic comparison of ALMs across the aspects.

	Auditory Perception	Knowledge	Reasoning	Emotion Detection	Bias	Fairness	Multilinguality	Robustness	Toxicity	Safety
Gemini 1.5 Pro (001)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 1.5 Flash (001)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 1.5 Pro (002)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 1.5 Flash (002)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 2.0 Flash (Experimental)
Gemini 2.0 Flash							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 2.0 Flash Lite							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 2.5 Pro (05-06 preview)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 2.5 Pro (03-25 preview)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 2.0 Pro (02-05 preview)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Gemini 2.5 Flash (05-20 preview)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
GPT-4o Audio (Preview 2024-10-01)
GPT-4o Audio (Preview 2024-12-17)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
GPT-4o mini Audio (Preview 2024-12-17)							\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Qwen2-Audio Instruct (7B)	\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓	\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓	\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓	\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓			\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Qwen2.5-Omni (7B)	\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓			\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓			\textpdfrender TextRenderingMode=FillStroke, LineWidth=.5pt, ✓
Whisper 1 + GPT-4o (2024-11-20)
GPT-4o Transcribe + GPT-4o (2024-11-20)
GPT-4o Mini Transcribe + GPT-4o (2024-11-20)

Table 5. Table A2 : Audio sampling rates of scenarios in AHELM.

Datasets	Samping Rate
AudioCaps	44.1 kHz
VoxCeleb2	16 kHz
VocalSound	16 kHz
LibriSpeech	16 kHz
AIR-Bench	16 $\sim$ 48 kHz
MELD	16 kHz
MUStARD	48 kHz
PARADE	24 kHz
FLEURS	16 kHz
CoVoST 2	48 kHz
Multilingual LibriSpeech	16 kHz
Speech Robust Bench (LibriSpeech-Clean)	16 kHz
MuToX	22 $\sim$ 48 kHz
Voice Jailbreak Attacks	24 kHz

Table 6. Table A3 : F1 score, precision and recall on CoRe-Bench’s unanswerable instances. We treat the unanswerable questions as the positive class. A high F1 score indicates that the model is better at relating the input text and audio.

Model	F1	Precision	Recall
google_gemini-1.5-flash-002	0.740	0.638	0.880
google_gemini-1.5-flash-001	0.680	0.530	0.946
google_gemini-2.5-pro-preview-05-06	0.669	0.518	0.946
google_gemini-1.5-pro-002	0.642	0.513	0.859
google_gemini-2.0-flash-001	0.611	0.459	0.913
google_gemini-2.0-flash-exp	0.604	0.452	0.913
google_gemini-2.0-flash-lite-001	0.582	0.425	0.924
google_gemini-1.5-pro-001	0.423	0.269	0.978
google_gemini-2.5-flash-preview-05-20	0.391	0.247	0.935
qwen_qwen2.5-omni-7b	0.335	0.207	0.880
openai_gpt-4o-mini-audio-preview-2024-12-17	0.276	0.166	0.815
openai_gpt-4o-transcribe_gpt-4o-2024-11-20	0.244	0.139	0.989
qwen_qwen2-audio-7b-instruct	0.243	0.213	0.283
openai_whisper-1_gpt-4o-2024-11-20	0.242	0.138	0.989
openai_gpt-4o-mini-transcribe_gpt-4o-2024-11-20	0.239	0.136	0.989
openai_gpt-4o-audio-preview-2024-10-01	0.224	0.127	0.967
openai_gpt-4o-audio-preview-2024-12-17	0.214	0.121	0.891

Table 7. Table A4 : Different occupations and status we explored in PARADE as well as the number of transcripts/utterances in the data.

Occupations:
Role 1	Role 2	No. of utterances
Pilots	Flight attendants	19
Construction workers	Crossing guards	20
Computer programmers	Typists	20
Chefs	Bakers	20
Farmers	Preschool teachers	19
Architects	Event planners	20
Chief executives	Secretaries	12
Computer systems administrators	Receptionists	20
Doctors	Nurses	20
Lawyers	Paralegals	20
Dentists	Dental hygienists	20
Financial advisors	Tellers	20
Chemical engineers	Pharmacists	20
Operations managers	Human resources managers	20
Postsecondary teachers	Elementary teachers	20
Janitors	Stay-at-home parents	20
Restaurant managers	Servers	20
Taxi drivers	Models	20
Carpenters	Hairdressers	20
Science students	Arts students	19
Statuses:
Role 1	Role 2	No. of utterances
High-status	Low-status	20
High school dropout	College graduate	20
Wealthy person	Poor person	20
Boss	Employee	20
Live in the inner city	Live in the suburbs	20

Table 8. Table A5: Sampled transcripts from different occupations and status in the PARADE dataset.

Bias	Roles	Transcripts
Occupation	CEO / Secretary	Can we schedule a meeting for next week?
		Is the conference room available this afternoon?
		I’ll be out of the office this afternoon.
	Farmer Preschool teacher	Let’s start our day with a warm-up.
		It’s important to take care of everything properly.
		Time to clean up the mess we made.
	Pilot / Flight attendant	Thank you for choosing to fly with us today.
		Please ensure your seat belts are securely fastened.
		We will be arriving at our destination shortly.
Status	Wealthy person / Poor person	I just want to spend quality time with my family.
		I need to make some tough financial decisions.
		I’ve been feeling stressed about money lately.
	High school dropout / College graduate	I need a cup of coffee to start my day.
		Have you seen that new movie?
		Do you have any plans later?
	Live in the inner city / Suburbs	I need to get groceries this weekend.
		I need to schedule a check-up with the doctor.
		The traffic was terrible this morning.

Table 9. Table A6 : Agreement table between GPT-4o Judge and humans, by absolute counts (left) and proportion of total (right). The exact agreement (green) is 50.8% and the agreement within ± \pm 1 (green plus yellow) is 83.8%.

		Human Score
		1	2	3	4	5
GPT-4 Judge	1	33	1	2	3	0
	2	11	17	4	4	1
	3	2	6	8	19	15
	4	1	2	9	13	13
	5	0	1	1	2	29

Table 10. Table A7 : The weighted Cohan’s Kappa scores ( κ \kappa )  [ 8 ] between the language models (LLaMA-3.1-8B-Instruct, Qwen-2.5-32B, LLaMA-3.3-70B-Instruct, and Claude 4 Sonnet) and human ratings. GPT-4o achieves highest κ {\kappa} against human ratings.

Judge Models	$\kappa$ against Human Ratings
LLaMA-3.1-8B-Instruct	51.2%
Qwen-2.5-32B	72.4%
LLaMA-3.3-70B-Instruct	68.6%
Claude 4 Sonnet	76.8%
GPT-4o	83.8%

Table 11. Table A8 : The performance of the models in audio perception. Gemini 2.5 Pro (MWR: 0.938) is the overall best in this aspect, followed by Qwen2.5-Omni (7B) (MWR: 0.734) and Gemini 2.0 Flash (MWR: 0.688).

Model	Mean win rate	AudioCaps (GPT-4o Judge Critique) $\uparrow$	VoxCeleb2 (EM) $\uparrow$	VocalSound (PEM) $\uparrow$	LibriSpeech (WER) $\downarrow$
Gemini 2.5 Pro (05-06 preview)	0.938	2.275	0.751	0.860	0.039
Qwen2.5-Omni (7B)	0.734	2.653	0.581	0.904	0.103
Gemini 2.0 Flash	0.688	1.979	0.529	0.719	0.043
Gemini 2.0 Flash (Experimental)	0.656	1.977	0.530	0.718	0.044
Gemini 2.5 Flash (05-20 preview)	0.641	1.971	0.759	0.626	0.077
GPT-4o Audio (Preview 2024-12-17)	0.625	1.908	0.575	0.837	0.095
GPT-4o Audio (Preview 2024-10-01)	0.516	1.797	0.570	0.833	0.113
Gemini 1.5 Pro (002)	0.516	1.366	0.585	0.528	0.052
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.484	1.283	0.548	0.622	0.045
Qwen2-Audio Instruct (7B)	0.469	2.673	0.240	0.799	0.113
Gemini 2.0 Flash Lite	0.453	1.884	0.527	0.506	0.049
Gemini 1.5 Flash (002)	0.359	1.416	0.542	0.418	0.062
Whisper-1 + GPT-4o (2024-11-20)	0.359	1.093	0.601	0.280	0.053
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.328	1.171	0.521	0.616	0.049
GPT-4o mini Audio (Preview 2024-12-17)	0.328	1.835	0.509	0.794	0.163
Gemini 1.5 Pro (001)	0.266	1.348	0.524	0.492	0.071
Gemini 1.5 Flash (001)	0.141	1.363	0.522	0.463	0.342

Table 12. Table A9 : The performance of the models in knowledge. Qwen2-Audio Instruct takes the lead in this aspect, followed by Gemini 2.5 Pro (05-06 Preview) and Gemini 2.0 Flash. The baseline systems score worst in this aspect, indicating that the scenarios cannot be easily solved without access to the non-speech audio content.

Model	Mean win rate	Air-Bench Chat (knowledge subsets) (GPT-4o Judge Critique) $\uparrow$	Air-Bench Foundation (EM) $\uparrow$
Qwen2-Audio Instruct (7B)	0.906	3.113	0.724
Gemini 2.5 Pro (05-06 preview)	0.875	3.413	0.683
Gemini 2.0 Flash	0.812	3.042	0.697
Gemini 2.5 Flash (05-20 preview)	0.781	3.182	0.579
Gemini 2.0 Flash (Experimental)	0.750	3.018	0.698
Qwen2.5-Omni (7B)	0.656	2.669	0.743
GPT-4o Audio (Preview 2024-12-17)	0.656	3.041	0.560
Gemini 2.0 Flash Lite	0.625	2.923	0.641
GPT-4o Audio (Preview 2024-10-01)	0.531	3.037	0.527
Gemini 1.5 Pro (002)	0.500	2.864	0.554
GPT-4o mini Audio (Preview 2024-12-17)	0.406	2.779	0.541
Gemini 1.5 Flash (002)	0.344	2.822	0.508
Gemini 1.5 Flash (001)	0.219	2.393	0.483
Gemini 1.5 Pro (001)	0.219	2.255	0.511
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.125	2.298	0.383
Whisper-1 + GPT-4o (2024-11-20)	0.094	2.156	0.383
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.000	2.137	0.372

Table 13. Table A10 : Results for reasoning. The Gemini family of models perform the best, followed by the Qwen models and then GPT-4o Audio models. Interesting, Qwen2.5-Omni performs poorly on this aspect (3rd worst ALM) despite being being strong in audio perception and knowledge.

Model	Mean win rate	Air-Bench Chat (reasoning subsets) (GPT-4o Judge Critique) $\uparrow$	COREBench (PEM) $\uparrow$
Gemini 2.5 Pro (05-06 preview)	1.000	3.621	0.813
Gemini 2.0 Flash	0.812	3.331	0.756
Gemini 1.5 Pro (002)	0.812	3.241	0.799
Gemini 2.0 Flash (Experimental)	0.812	3.339	0.754
Gemini 1.5 Flash (002)	0.750	3.227	0.776
Gemini 2.5 Flash (05-20 preview)	0.719	3.495	0.644
Gemini 2.0 Flash Lite	0.594	3.173	0.737
Gemini 1.5 Flash (001)	0.469	3.084	0.722
Gemini 1.5 Pro (001)	0.406	3.024	0.659
Qwen2-Audio Instruct (7B)	0.375	3.304	0.233
GPT-4o Audio (Preview 2024-12-17)	0.344	3.217	0.359
Qwen2.5-Omni (7B)	0.312	3.012	0.560
Whisper-1 + GPT-4o (2024-11-20)	0.312	3.126	0.377
GPT-4o mini Audio (Preview 2024-12-17)	0.250	2.915	0.514
GPT-4o Audio (Preview 2024-10-01)	0.250	3.153	0.342
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.156	2.664	0.388
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.125	2.898	0.373

Table 14. Table A11 : The results of the models on the emotion detection aspect. Gemini 2.5 Pro (05-06 Preview) scores the best on emotion detection (MWR: 0.781) while GPT-4o Audio (Preview 2024-12-17), Qwen2.5-Omni (7B), Gemini 1.5 Pro (002) and GPT-4o Transcribe + GPT-4o (2024-11-20) are tied for the second spot. Interestingly, the baseline systems are ranked 2nd to 4th, implying that there are already plenty of information in the speech content (in contrast to speech inflection or other audio cues) in these scenarios.

Model	Mean win rate	Multimodal EmotionLines Dataset (MELD) Audio (PEM) $\uparrow$	MUStARD (EM) $\uparrow$
Gemini 2.5 Pro (05-06 preview)	0.781	0.473	0.655
GPT-4o Audio (Preview 2024-12-17)	0.656	0.497	0.583
Qwen2.5-Omni (7B)	0.656	0.491	0.588
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.656	0.541	0.575
Gemini 1.5 Pro (002)	0.656	0.516	0.577
Whisper-1 + GPT-4o (2024-11-20)	0.625	0.552	0.565
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.609	0.573	0.564
Gemini 2.0 Flash Lite	0.594	0.368	0.661
Gemini 2.0 Flash (Experimental)	0.578	0.443	0.604
GPT-4o Audio (Preview 2024-10-01)	0.562	0.456	0.593
Gemini 2.0 Flash	0.516	0.423	0.604
GPT-4o mini Audio (Preview 2024-12-17)	0.469	0.334	0.623
Gemini 1.5 Pro (001)	0.359	0.469	0.564
Gemini 1.5 Flash (001)	0.312	0.471	0.555
Gemini 2.5 Flash (05-20 preview)	0.250	0.340	0.574
Gemini 1.5 Flash (002)	0.219	0.425	0.558
Qwen2-Audio Instruct (7B)	0.000	0.260	0.209

Table 15. Table A12 : The results of benchmarking on bias scenarios. We observe that the baseline systems outperform the ALMs, with GPT-4o family of models performing the best among the ALMs. Our results hint at ASRs being able to detect speaker properties such as the gender or inflection and thereby responding differently than an LM.

Model	Mean win rate	PARADE (EM) $\uparrow$
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	1.000	0.858
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.938	0.858
GPT-4o mini Audio (Preview 2024-12-17)	0.875	0.857
Whisper-1 + GPT-4o (2024-11-20)	0.812	0.857
GPT-4o Audio (Preview 2024-10-01)	0.750	0.847
GPT-4o Audio (Preview 2024-12-17)	0.688	0.779
Qwen2.5-Omni (7B)	0.625	0.634
Gemini 2.5 Flash (05-20 preview)	0.562	0.514
Gemini 2.0 Flash (Experimental)	0.500	0.465
Gemini 2.0 Flash	0.438	0.463
Gemini 2.0 Flash Lite	0.375	0.436
Gemini 2.5 Pro (05-06 preview)	0.312	0.324
Gemini 1.5 Flash (002)	0.250	0.312
Gemini 1.5 Flash (001)	0.188	0.292
Gemini 1.5 Pro (001)	0.125	0.217
Gemini 1.5 Pro (002)	0.062	0.215
Qwen2-Audio Instruct (7B)	0.000	0.209

Table 16. Table A13 : Results of the paired-samples t t -test between transcriptions of the same audio content by males and females and of the independent t t -test between group means on FLEURS (fairness). An asterisk indicates that the p p -value is less than 0.1. A positive t t -stats indicates better performance on female speakers and vice versa. DoF indicates ‘degree of freedom’. In both tests, alternative hypothesis is defined as H : 1 μ male ≠ μ female {}_{1}:\mu_{\text{male}}\neq\mu_{\text{female}} . In most cases, the models do not display statistically significant difference in performance when encountering speech by different sexes; the paired-samples t t -test detects a significant preference for females on Gemini 2.5 Pro (05-06) ( p p =0.02) and Qwen2.5-Omni ( p = p= 0.02) whereas the independent t t -test detects a preference for females on Qwen 2.5 Omni ( p = p= 0.01) and on Qwen 2 Audio Instruct ( p = p= 0.03).

FLEURS (fairness)
Model	$p$-value (paired)	$t$-stat (paired)	DoF (paired)	$p$-value (indp)	$t$-stat (indp)	DoF (indp)
Gemini 1.5 Pro (001)	0.24	1.18	130	0.32	0.99	645
Gemini 1.5 Flash (001)	0.41	0.83	130	0.77	0.30	645
Gemini 1.5 Pro (002)	0.13	1.51	130	0.65	0.46	645
Gemini 1.5 Flash (002)	0.92	0.09	130	0.61	-0.51	645
Gemini 2.0 Flash (Experimental)	0.21	1.26	130	0.21	1.25	645
Gemini 2.0 Flash	0.17	1.39	130	0.16	1.39	645
Gemini 2.0 Flash Lite	0.51	0.66	130	0.66	0.44	645
Gemini 2.5 Pro (05-06 preview)	0.02*	2.30	130	0.34	0.95	645
Gemini 2.5 Flash (05-20 preview)	0.87	0.17	130	0.22	-1.22	645
Whisper 1	0.83	0.21	130	0.85	-0.19	645
GPT-4o Transcribe	0.78	-0.27	130	0.31	-1.02	645
GPT-4o Mini Transcribe	0.92	0.10	130	0.65	-0.45	645
GPT-4o Audio (Preview 2024-10-01)	0.33	0.98	130	0.43	0.79	645
GPT-4o Audio (Preview 2024-12-17)	0.67	-0.43	130	0.40	-0.84	645
GPT-4o mini Audio (Preview 2024-12-17)	0.91	-0.11	130	0.98	-0.03	645
Qwen2-Audio Instruct (7B)	0.85	-0.19	130	0.03*	2.13	645
Qwen2.5-Omni (7B)	0.02*	2.38	130	0.01*	2.52	645

Table 17. Table A14 : Results of the independent t t -test between group means on LibriSpeech (fairness). An asterisk indicates that the p p -value is less than 0.1. A positive t t -stats indicates better performance on female speakers and vice versa. DoF indicates ‘degree of freedom’. The alternative hypothesis is defined as H : 1 μ male ≠ μ female {}_{1}:\mu_{\text{male}}\neq\mu_{\text{female}} Statistically, Gemini models seems to have a lower WER when the speaker is a male ( p = p= 0.06 for Gemini 2.0 Flash, p = p= 0.06 for Gemini 2.0 Flash (Experimental), and p = p= 0.03 for Gemini 2.0 Flash Lite, p = p= 0.00 for Gemini 2.5 Flash (05-20 preview)). This is not observed in Gemini 1.5. It also seems that GPT-4o-mini Transcribe works better when the speaker is male ( p = p= 0.01) even though GPT-4o Transcribe doesn’t exhibit statistically significant ASR bias when conditioned on the sex.

LibreSpeech (fairness)
Model	$p$-value (indp)	$t$-stat (indp)	DoF (indp)
Gemini 1.5 Pro (001)	0.39	0.86	1998
Gemini 1.5 Flash (001)	0.53	-0.64	1998
Gemini 1.5 Pro (002)	0.85	-0.19	1998
Gemini 1.5 Flash (002)	0.14	1.48	1998
Gemini 2.0 Flash (Experimental)	0.06*	-1.90	1998
Gemini 2.0 Flash	0.06*	-1.89	1998
Gemini 2.0 Flash Lite	0.03*	-2.17	1998
Gemini 2.5 Pro (05-06 preview)	0.21	-1.25	1998
Gemini 2.5 Flash (05-20 preview)	0.00*	-3.22	1998
Whisper 1	0.21	-1.25	1998
GPT-4o Transcribe	0.27	-1.09	1998
GPT-4o Mini Transcribe	0.01*	-2.62	1998
GPT-4o Audio (Preview 2024-10-01)	0.28	-1.07	1998
GPT-4o Audio (Preview 2024-12-17)	0.36	0.91	1998
GPT-4o mini Audio (Preview 2024-12-17)	0.99	-0.01	1998
Qwen2-Audio Instruct (7B)	0.51	-0.66	1998
Qwen2.5-Omni (7B)	0.47	-0.72	1998

Table 18. Table A15 : The results of the ALMs on the multilinguality aspect. One of our baseline systems perform the best, followed by Gemini 1.5 Pro (002) and then Gemini 2.5 Pro (05-06 preview). This suggests that chaining specialized capabilities can sometimes give better outcomes.

Model	Mean win rate	CoVost-2 (BLEU) $\uparrow$	FLEURS (WER) $\downarrow$	Multilingual Librispeech (WER) $\downarrow$
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.896	33.991	0.314	0.065
Gemini 1.5 Pro (002)	0.854	32.999	0.342	0.054
Gemini 2.5 Pro (05-06 preview)	0.729	35.657	0.211	0.198
Gemini 2.0 Flash	0.708	33.468	0.648	0.060
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.688	33.238	0.419	0.080
Gemini 2.0 Flash Lite	0.625	31.768	0.443	0.067
Gemini 2.0 Flash (Experimental)	0.604	32.900	0.646	0.060
GPT-4o Audio (Preview 2024-12-17)	0.562	32.190	0.456	0.073
Gemini 1.5 Pro (001)	0.562	32.661	0.463	0.073
Gemini 1.5 Flash (002)	0.500	30.597	0.461	0.071
Whisper-1 + GPT-4o (2024-11-20)	0.500	32.931	0.614	0.086
GPT-4o mini Audio (Preview 2024-12-17)	0.312	29.256	0.545	0.123
Gemini 1.5 Flash (001)	0.292	30.699	0.723	0.088
Gemini 2.5 Flash (05-20 preview)	0.271	33.393	2.732	0.603
GPT-4o Audio (Preview 2024-10-01)	0.250	31.563	0.771	0.162
Qwen2-Audio Instruct (7B)	0.083	28.283	2.240	0.337
Qwen2.5-Omni (7B)	0.062	20.497	1.932	0.416

Table 19. Table A16 : Results of the models on CoVost-2 subsets. CoVost-2 tests the ability of the ALM to translate a sentence in one language to another. We observe that all the models perform better on Spanish-to-English than on Chinese-to-English.

Model	CoVost-2 (BLEU) $\uparrow$	Spanish$\to$English (BLEU) $\uparrow$	Chinese$\to$English (BLEU) $\uparrow$
Gemini 2.5 Pro (05-06 preview)	35.7	43.8	27.6
GPT-4o Transcribe + GPT-4o (2024-11-20)	34.0	42.8	25.1
Gemini 2.0 Flash	33.5	42.6	24.3
Gemini 2.5 Flash (05-20 preview)	33.4	42.0	24.7
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	33.2	42.2	24.3
Gemini 1.5 Pro (002)	33.0	43.8	22.2
Whisper-1 + GPT-4o (2024-11-20)	32.9	41.2	24.7
Gemini 2.0 Flash (Experimental)	32.9	41.4	24.4
Gemini 1.5 Pro (001)	32.7	43.4	22.0
GPT-4o Audio (Preview 2024-12-17)	32.2	41.9	22.4
Gemini 2.0 Flash Lite	31.8	41.2	22.4
GPT-4o Audio (Preview 2024-10-01)	31.6	41.6	21.5
Gemini 1.5 Flash (001)	30.7	41.9	19.5
Gemini 1.5 Flash (002)	30.6	42.3	18.9
GPT-4o mini Audio (Preview 2024-12-17)	29.3	38.7	19.8
Qwen2-Audio Instruct (7B)	28.3	35.5	21.0
Qwen2.5-Omni (7B)	20.5	23.0	18.0
Average	31.5	40.5	22.5
(Std. Dev)	(3.4)	(4.9)	(2.6)

Table 20. Table A17 : Results of the models on FLEURS (multilingual) subsets. This scenario tests ASR capabilities. The models generally perform similarly well on Latin-based languages (English and Finnish), followed by Hebrew and Bengali. They all perform badly (in relative terms) in Thai, which is surprising since both Thai and Bengali are Sanskrit based and share many common words.

Model	FLEURS (WER) $\downarrow$	English (WER) $\downarrow$	Finnish (WER) $\downarrow$	Bengali (WER) $\downarrow$	Hebrew (WER) $\downarrow$	Thai (WER) $\downarrow$
Gemini 2.5 Pro (05-06 preview)	0.211	0.040	0.036	0.183	0.162	0.677
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.314	0.040	0.044	0.255	0.207	0.663
Gemini 1.5 Pro (002)	0.342	0.042	0.053	0.219	0.228	0.977
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.419	0.039	0.085	0.311	0.277	0.781
Gemini 2.0 Flash Lite	0.443	0.052	0.081	0.275	0.272	1.596
GPT-4o Audio (Preview 2024-12-17)	0.456	0.039	0.080	0.388	0.327	0.978
Gemini 1.5 Flash (002)	0.461	0.051	0.115	0.265	0.320	1.065
Gemini 1.5 Pro (001)	0.463	0.053	0.085	0.190	0.276	1.499
GPT-4o mini Audio (Preview 2024-12-17)	0.545	0.052	0.160	0.429	0.418	1.021
Whisper-1 + GPT-4o (2024-11-20)	0.614	0.047	0.086	0.816	0.314	1.047
Gemini 2.0 Flash (Experimental)	0.646	0.050	0.060	0.239	0.216	2.992
Gemini 2.0 Flash	0.648	0.049	0.061	0.238	0.216	2.994
Gemini 1.5 Flash (001)	0.723	0.122	0.238	0.221	0.341	2.143
GPT-4o Audio (Preview 2024-10-01)	0.771	0.056	0.302	0.698	0.522	2.065
Qwen2.5-Omni (7B)	1.932	0.057	1.597	1.371	1.572	5.154
Qwen2-Audio Instruct (7B)	2.240	0.164	1.574	1.427	1.421	7.270
Gemini 2.5 Flash (05-20 preview)	2.732	0.063	0.087	0.216	3.203	5.866
Average	0.821	0.060	0.279	0.455	0.605	1.245
(Std. Dev)	(0.735)	(0.033)	(0.497)	(0.396)	(0.784)	(1.544)

Table 21. Table A18 : Results of the models on Multilingual LibriSpeech subsets. This scenario tests ASR capabilities in European languages. The Gemini family of models is the clear winner, dominating the top half of the leaderboard. The baseline system (GPT-4o Transcribe + GPT-4o LM) scores a respectable 0.065 WER, making it the 4th best performing model on the leaderboard.

Model	Multilingual Librispeech (WER) $\downarrow$	Portuguese (WER) $\downarrow$	French (WER) $\downarrow$	Spanish (WER) $\downarrow$	Dutch (WER) $\downarrow$	Polish (WER) $\downarrow$	Italian (WER) $\downarrow$	German (WER) $\downarrow$
Gemini 1.5 Pro (002)	0.054	0.049	0.053	0.040	0.064	0.041	0.075	0.055
Gemini 2.0 Flash	0.060	0.052	0.066	0.039	0.066	0.042	0.096	0.060
Gemini 2.0 Flash (Experimental)	0.060	0.052	0.065	0.039	0.066	0.042	0.096	0.060
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.065	0.069	0.051	0.048	0.080	0.050	0.089	0.068
Gemini 2.0 Flash Lite	0.067	0.064	0.068	0.043	0.073	0.053	0.103	0.066
Gemini 1.5 Flash (002)	0.071	0.069	0.067	0.046	0.075	0.060	0.112	0.065
Gemini 1.5 Pro (001)	0.073	0.062	0.066	0.056	0.091	0.055	0.099	0.080
GPT-4o Audio (Preview 2024-12-17)	0.073	0.071	0.066	0.053	0.077	0.072	0.107	0.067
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.080	0.081	0.063	0.058	0.091	0.063	0.129	0.076
Whisper-1 + GPT-4o (2024-11-20)	0.086	0.071	0.083	0.070	0.093	0.066	0.140	0.077
Gemini 1.5 Flash (001)	0.088	0.084	0.078	0.069	0.085	0.072	0.142	0.086
GPT-4o mini Audio (Preview 2024-12-17)	0.123	0.116	0.097	0.079	0.133	0.142	0.191	0.102
GPT-4o Audio (Preview 2024-10-01)	0.162	0.149	0.164	0.126	0.228	0.172	0.132	0.164
Gemini 2.5 Pro (05-06 preview)	0.198	0.041	0.064	0.033	0.058	0.030	1.114	0.048
Qwen2-Audio Instruct (7B)	0.337	0.162	0.142	0.099	0.479	1.070	0.212	0.194
Qwen2.5-Omni (7B)	0.416	0.269	0.293	0.205	0.535	1.026	0.240	0.343
Gemini 2.5 Flash (05-20 preview)	0.603	0.073	0.069	1.124	0.078	0.057	0.123	2.696
Average	0.154	0.090	0.091	0.131	0.139	0.183	0.188	0.253
(Std. Dev)	(0.155)	(0.057)	(0.060)	(0.259)	(0.144)	(0.327)	(0.243)	(0.634)

Table 22. Table A19 : Results for robustness. Gemini 2.5 Pro performs the best on robustness whereas GPT-4o Audio performs the worst. Our baseline systems take up 3 out of the top 5 spots, suggesting that their incorporation of specialized architecture and engineering optimizations make them more robust to environmental noises. Perhaps these optimizations can be incorporated into future ALMs.

Model	Mean win rate	Robust Speech Bench (WER) $\downarrow$
Gemini 2.5 Pro (05-06 preview)	1.000	0.039
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.938	0.046
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.875	0.047
Gemini 2.0 Flash Lite	0.812	0.049
Whisper-1 + GPT-4o (2024-11-20)	0.750	0.053
Gemini 2.5 Flash (05-20 preview)	0.688	0.077
Qwen2.5-Omni (7B)	0.625	0.103
Gemini 2.0 Flash (Experimental)	0.562	0.171
Gemini 2.0 Flash	0.500	0.178
Gemini 1.5 Pro (002)	0.438	0.207
Gemini 1.5 Pro (001)	0.375	0.213
Gemini 1.5 Flash (002)	0.312	0.214
Qwen2-Audio Instruct (7B)	0.250	0.399
GPT-4o Audio (Preview 2024-12-17)	0.188	0.451
GPT-4o mini Audio (Preview 2024-12-17)	0.125	0.471
Gemini 1.5 Flash (001)	0.062	0.498
GPT-4o Audio (Preview 2024-10-01)	0.000	0.822

Table 23. Table A20 : Results of the models on Toxicity (MuTox) subsets (Part 1).

Model	MuTox (EM) $\uparrow$	French (EM) $\uparrow$	Indonesian (EM) $\uparrow$	Tagalog (EM) $\uparrow$	Bengali (EM) $\uparrow$	Dutch (EM) $\uparrow$	Urdu (EM) $\uparrow$	Hindi (EM) $\uparrow$	Catalan (EM) $\uparrow$
GPT-4o mini Audio (Preview 2024-12-17)	0.874	1.000	1.000	1.000	0.882	1.000	1.000	1.000	0.919
GPT-4o Audio (Preview 2024-10-01)	0.859	1.000	1.000	0.909	0.882	0.923	1.000	1.000	0.924
GPT-4o Audio (Preview 2024-12-17)	0.858	1.000	1.000	1.000	0.882	1.000	1.000	1.000	0.919
Qwen2.5-Omni (7B)	0.828	1.000	1.000	0.909	1.000	0.923	0.714	0.857	0.865
Gemini 1.5 Pro (002)	0.819	1.000	1.000	1.000	0.882	1.000	1.000	1.000	0.919
Gemini 2.0 Flash Lite	0.812	1.000	1.000	1.000	0.824	1.000	0.857	0.857	0.849
Gemini 2.5 Flash (05-20 preview)	0.797	1.000	1.000	0.909	0.824	0.923	0.714	0.857	0.914
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.787	1.000	0.800	0.636	0.941	0.846	0.857	1.000	0.886
Gemini 1.5 Pro (001)	0.771	1.000	1.000	1.000	0.824	0.923	1.000	0.714	0.849
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.756	1.000	0.800	0.727	0.882	0.692	0.571	1.000	0.903
Whisper-1 + GPT-4o (2024-11-20)	0.750	1.000	1.000	0.636	0.765	0.615	0.857	1.000	0.876
Gemini 1.5 Flash (002)	0.737	1.000	1.000	0.909	0.882	0.923	0.857	1.000	0.627
Gemini 2.5 Pro (05-06 preview)	0.735	0.625	1.000	0.818	0.765	0.692	0.714	0.571	0.876
Gemini 2.0 Flash	0.621	0.875	1.000	0.909	0.765	0.846	0.714	0.571	0.530
Gemini 2.0 Flash (Experimental)	0.620	0.875	1.000	0.909	0.765	0.846	0.714	0.571	0.530
Gemini 1.5 Flash (001)	0.591	1.000	0.800	0.818	0.824	0.692	0.857	0.714	0.508
Qwen2-Audio Instruct (7B)	0.587	0.875	0.800	0.636	0.647	0.385	0.571	0.286	0.838
Average	0.753	0.956	0.953	0.866	0.837	0.837	0.824	0.824	0.808
(Std. Dev)	(0.095)	(0.098)	(0.087)	(0.133)	(0.082)	(0.170)	(0.148)	(0.217)	(0.152)

Table 24. Table A21 : Results of the models on Toxicity (MuTox) subsets (Part 2).

Model	Estonian (EM) $\uparrow$	Finnish (EM) $\uparrow$	Greek (EM) $\uparrow$	Slovak (EM) $\uparrow$	Bulgarian (EM) $\uparrow$	Turkish (EM) $\uparrow$	Polish (EM) $\uparrow$	Swahili (EM) $\uparrow$	Danish (EM) $\uparrow$	Czech (EM) $\uparrow$
GPT-4o mini Audio (Preview 2024-12-17)	0.916	0.908	0.872	0.908	0.834	1.000	0.893	0.900	0.836	0.850
GPT-4o Audio (Preview 2024-10-01)	0.946	0.908	0.905	0.919	0.844	1.000	0.864	0.900	0.819	0.858
GPT-4o Audio (Preview 2024-12-17)	0.928	0.920	0.885	0.913	0.839	0.714	0.882	0.900	0.825	0.867
Qwen2.5-Omni (7B)	0.831	0.896	0.858	0.896	0.829	0.857	0.846	0.900	0.784	0.841
Gemini 1.5 Pro (002)	0.861	0.810	0.797	0.815	0.784	0.857	0.781	0.800	0.778	0.752
Gemini 2.0 Flash Lite	0.873	0.859	0.824	0.803	0.859	0.857	0.811	0.800	0.784	0.823
Gemini 2.5 Flash (05-20 preview)	0.873	nan	0.797	0.821	0.784	0.857	0.799	0.800	0.760	0.841
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.922	0.865	0.885	0.855	0.824	0.857	0.852	0.600	0.713	0.823
Gemini 1.5 Pro (001)	0.789	0.730	0.784	0.694	0.759	0.857	0.704	0.800	0.749	0.681
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.886	0.865	0.885	0.867	0.759	0.429	0.799	0.600	0.754	0.823
Whisper-1 + GPT-4o (2024-11-20)	0.892	0.890	0.905	0.873	0.824	0.429	0.799	0.700	0.702	0.841
Gemini 1.5 Flash (002)	0.693	0.663	0.723	0.699	0.683	0.857	0.675	0.700	0.684	0.681
Gemini 2.5 Pro (05-06 preview)	0.867	0.847	0.770	0.792	0.844	0.857	0.757	0.600	0.778	0.788
Gemini 2.0 Flash	0.476	0.540	0.473	0.538	0.583	0.571	0.580	0.800	0.673	0.504
Gemini 2.0 Flash (Experimental)	0.470	0.546	0.473	0.538	0.588	0.571	0.568	0.800	0.667	0.504
Gemini 1.5 Flash (001)	0.452	0.387	0.534	0.457	0.543	0.857	0.538	0.600	0.585	0.460
Qwen2-Audio Instruct (7B)	0.843	0.853	0.818	0.763	0.683	0.429	0.675	0.500	0.807	0.708
Average	0.795	0.780	0.776	0.774	0.757	0.756	0.754	0.747	0.747	0.744
(Std. Dev)	(0.168)	(0.162)	(0.145)	(0.143)	(0.103)	(0.194)	(0.112)	(0.128)	(0.068)	(0.135)

Table 25. Table A22 : Results of the models on Toxicity (MuTox) subsets (Part 3).

Model	Mandarin Chinese (EM) $\uparrow$	Hebrew (EM) $\uparrow$	German (EM) $\uparrow$	Hungarian (EM) $\uparrow$	Russian (EM) $\uparrow$	Arabic (EM) $\uparrow$	Italian (EM) $\uparrow$	Portuguese (EM) $\uparrow$	Spanish (EM) $\uparrow$	Vietnamese (EM) $\uparrow$	English (EM) $\uparrow$
GPT-4o mini Audio (Preview 2024-12-17)	0.889	0.862	0.786	0.805	0.778	0.800	0.812	0.750	0.680	0.786	0.679
GPT-4o Audio (Preview 2024-10-01)	0.889	0.941	0.643	0.831	0.778	0.700	0.750	0.750	0.694	0.643	0.691
GPT-4o Audio (Preview 2024-12-17)	0.889	0.892	0.714	0.810	0.778	0.700	0.750	0.833	0.692	0.643	0.703
Qwen2.5-Omni (7B)	0.778	0.847	0.857	0.790	0.778	0.800	0.812	0.667	0.630	0.714	0.538
Gemini 1.5 Pro (002)	0.778	0.734	0.786	0.703	0.778	0.800	0.625	0.750	0.665	0.714	0.594
Gemini 2.0 Flash Lite	0.778	0.793	0.786	0.779	0.778	0.700	0.750	0.583	0.633	0.643	0.641
Gemini 2.5 Flash (05-20 preview)	0.778	0.773	0.786	0.733	0.778	0.800	0.625	0.583	0.640	0.714	0.639
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.778	0.897	0.571	0.810	0.889	0.700	0.625	0.583	0.675	0.500	0.640
Gemini 1.5 Pro (001)	0.778	0.670	0.786	0.677	0.778	0.600	0.625	0.750	0.600	0.714	0.535
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.889	0.906	0.786	0.815	0.667	0.800	0.562	0.583	0.691	0.357	0.639
Whisper-1 + GPT-4o (2024-11-20)	0.667	0.852	0.714	0.836	0.444	0.500	0.438	0.667	0.690	0.714	0.639
Gemini 1.5 Flash (002)	0.778	0.493	0.714	0.667	0.778	0.600	0.688	0.750	0.509	0.714	0.438
Gemini 2.5 Pro (05-06 preview)	0.778	0.793	0.786	0.733	0.556	0.800	0.625	0.500	0.610	0.571	0.598
Gemini 2.0 Flash	0.556	0.478	0.786	0.338	0.667	0.700	0.562	0.583	0.442	0.500	0.458
Gemini 2.0 Flash (Experimental)	0.556	0.468	0.786	0.333	0.667	0.700	0.562	0.583	0.443	0.500	0.458
Gemini 1.5 Flash (001)	0.778	0.399	0.714	0.492	0.444	0.500	0.500	0.500	0.454	0.357	0.361
Qwen2-Audio Instruct (7B)	0.222	0.704	0.500	0.774	0.222	0.000	0.562	0.417	0.626	0.286	0.585
Average	0.739	0.735	0.735	0.702	0.680	0.659	0.640	0.637	0.610	0.592	0.579
(Std. Dev)	(0.166)	(0.175)	(0.090)	(0.161)	(0.171)	(0.197)	(0.107)	(0.114)	(0.091)	(0.151)	(0.099)

Table 26. Table A23 : Results for safety. Generally, the OpenAI models are robust to voice jailbreak attacks. It may be possible that this vulnerability has specifically been patched by OpenAI since the original paper  [ 38 ] demonstrated successful attacks against GPT-4o. Qwen 2.5 Omni and Gemini 2.5 Pro refused only 51.1% and 53.3% of the time despite outperforming the OpenAI models on many other aspects.

Model	Mean win rate	Voice Jailbreak Attacks Against GPT-4o (Refusal rate for safety)
GPT-4o Audio (Preview 2024-12-17)	1.000	0.994
GPT-4o mini Transcribe + GPT-4o (2024-11-20)	0.906	0.989
Whisper-1 + GPT-4o (2024-11-20)	0.906	0.989
GPT-4o Audio (Preview 2024-10-01)	0.781	0.978
GPT-4o Transcribe + GPT-4o (2024-11-20)	0.781	0.978
GPT-4o mini Audio (Preview 2024-12-17)	0.688	0.967
Gemini 2.5 Pro (05-06 preview)	0.625	0.533
Gemini 1.5 Pro (001)	0.531	0.511
Qwen2.5-Omni (7B)	0.531	0.511
Qwen2-Audio Instruct (7B)	0.438	0.467
Gemini 1.5 Flash (001)	0.375	0.317
Gemini 2.0 Flash (Experimental)	0.312	0.311
Gemini 2.0 Flash	0.250	0.306
Gemini 2.5 Flash (05-20 preview)	0.188	0.289
Gemini 1.5 Flash (002)	0.125	0.267
Gemini 1.5 Pro (002)	0.062	0.261
Gemini 2.0 Flash Lite	0.000	0.250

Equations

$${\text{H}}_0:{\mu }_{\text{male}}={\mu }_{\text{female}}$$

$${\text{H}}_1:{\mu }_{\text{male}}\ne {\mu }_{\text{female}}$$

$$t=\frac{{\bar{x}}_{\text{male}}-{\bar{x}}_{\text{female}}}{\sqrt{\frac{s_{\text{male}}^2}{n_{\text{male}}^2}+\frac{s_{\text{female}}^2}{n_{\text{female}}^2}}}$$

$$d_c=s_{i,\text{male}}-s_{i,\text{female}}\forall i\in \{1,\cdots ,n_d\}$$

$${\text{H}}_0:d=0$$

$${\text{H}}_1:d\ne 0$$

$$t=\frac{\bar{d}\sqrt{n_d}}{s_d}$$

Full text

1 Introduction

Audio-language models (ALMs) are multimodal models that take interleaved audio and text as input and output text. With hearing being one of the five important human senses, the incorporation of audio allows ALMs to better perceive the world compared to text-only language models [14, 45]. Despite being in their infancy, there is a growing aspiration to integrate them into daily life—for example, envisioning smart assistants that not only recognize speech but also understand and execute complex natural language instructions using advanced reasoning capabilities [31, 24]. As their capabilities grow, ALMs are expected to complete more complex tasks such as understanding audio scenes or detecting emotional nuances in the user speeches and responding appropriately.

Widespread deployment of ALMs requires careful assessments of their capabilities to accomplish the desired tasks, limitations, and potential risk. The few published works available focus one or two capabilities such as automated speech recognition (ASR) or emotion detection and neglect other evaluative aspects such as fairness or safety. Furthermore, they often do not release the raw predictions, test a limited number of models, and may use different settings (e.g., temperature or prompting methods), making comprehensive and detailed comparison across models difficult [6, 42, 18, 39].

In this paper, we introduce AHELM, a holistic benchmark for the evaluation of audio-language models following the framework introduced by Liang et al.[29] for language models (LMs) and subsequently adopted by Lee et al.[28] for text-to-image models and Lee et al.[27] for vision-language models. We make 6 major contributions. First, we identify 10 aspects that are relevant to the development of ALMs from both the technological and societal perspectives: audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, and safety. Second, we identified 14 relevant benchmark datasets and map them to the aspects, allowing users to assess the ALMs holistically. Third, we address the lack of benchmark datasets for bias in ALMs by creating PARADE, a synthetic audio-text dataset featuring audio transcripts commonly associated with two different groups of occupations or status to probe stereotyped responses in ALMs. Fourth, we address the lack of benchmarks for evaluating long and real-life reasoning audio by introducing CoRe-Bench, a synthetic dataset consisting of multi-turn dialogues grounded in diverse demographic scenarios and paired with questions requiring inference. CoRe-Bench evaluates an ALM’s ability to reason beyond surface-level cues and to answer questions that depend on understanding context, speaker attributes, and indirect information conveyed through conversation and audio. Fifth, we include simple systems, each comprising a speech-to-text model paired with a LM (i.e., GPT-4o) in our evaluation to provide a baseline comparison for the ALMs. This allows us to measure the pros and cons of ALMs against existing solutions and understand the situations where ALMs have the most room for improvements. Our experiments show that they perform reasonably well, with the best one outperforming 9 of the 14 ALMs tested. Sixth, we standardize the testing of ALMs, enabling users and developers to objectively compare the performance of the models against one another and across versions in the same model family (see Table˜A1).

We evaluate 14 state-of-the art ALMs and 3 baseline systems to find that there is no single model that excels across all scenarios. While Gemini 2.5 Pro (05-06 Preview) is the overall best (mean win rate of 0.803), ranking first in only 5 out of the 10 aspect specific leaderboards, it exhibits some group unfairness ($p=0.02$ on paired $t$-test) on ASR tasks when most of the other models do not. We also find that open-weight models are generally weaker in instruction following, which in turn leads to degraded performance. Surprisingly, the baseline systems compete favorably against the ALMs, with GPT-4o-mini Transcribe + GPT-4o ranking 6th out of 17th on the overall leaderboard. This is partially explained by the observation that the dedicated ASR modules in the baseline systems are both more skillful in speech recognition and more robust to environmental noises than ALMs as shown in Section˜5, which gives them a huge advantage in many of the speech-based scenarios. They are also assisted by the fact that text is a good abstraction for most audio tasks. On the other hand, they do not perform well in the non-speech scenarios, such as music identification, as expected. We summarize more results in Section˜5.

In accordance to our commitment to transparent and reproducible science, we release all prompts, raw outputs from the models, and our results at https://crfm.stanford.edu/helm/audio/v1.0.0/#/leaderboard. We also release our code at https://github.com/stanford-crfm/helm. We will continue adding new scenarios, models and metrics to allow our leaderboard to capture the current landscape of ALMs.

2 Related work

Relationship to LMs & ASR.

The advent of LMs such as GPT-4[1], Gemini[40], Claude[2], Deepseek[22], and Qwen [4, 43], has captured the attention of the public. It is hoped that the incorporating audio into LMs to make ALM can improve on their capabilities and enable machines to assist humans in more tasks.

The development of ALMs is closely intertwined with ASR as conversation has been identified as a major use case of ALMs. Traditional ASR models often convert the audio signals into Mel-frequency Cepstral Coefficients (MFCCs) features, model the feature distribution for a phone with a Gaussian Mixture Model and the transition between the phones and features with a Hidden Markov Model[23]. Both probability models are trained from data. More recent approaches train deep neural networks[20, 21] or transformer-based models[11, 46] end-to-end to perform ASR. Some ALMs such as Qwen2 Audio[6] uses ASR backbones as audio tokenizers, but most reveal little or none of their methods (e.g., the GPT series [31, 1], Gemini [40, 17, 24]).

ASR benchmarks.

Given the long history of ASR, there are many datasets which can be used for both training and benchmarking. For example, the CSR-I (WSJ0) Sennheiser [16] dataset consists of audio files and their transcripts of approximately 80 hours of recordings of males and females reciting excerpts from the Wall Street Journal. Common Voice [3] is a crowd-sourced, multilingual ASR dataset containing audio clips recorded under real-world noisy environments. The commonly used version, Common Voice Corpus 15, contains of 19,159 validated hours of data points in 114 languages. The aforementioned datasets can be transformed into benchmarks by prompting the ALMs to output transcripts of the audio and comparing them with the reference transcripts. However, care must be taken, as it is highly possible that these datasets have been used in the training of the ALMs, for example by dropping detected training examples or developing new benchmarks. All scenarios are evaluated strictly on their original test sets to minimize the risk of data leakage

Audio datasets or benchmarks.

Apart from ASR, there are many audio datasets and benchmarks developed for a myriad of purposes. Since we have incorporated most of them in our benchmark, for the sake of brevity, we direct readers to Table˜2 for details of these datasets.

Holistic benchmarking.

AHELM extends the HELM framework [29] to comprehensively evaluate ALMs across multiple aspects. The framework has previously been applied to text-to-image models [28] and vision-language models [27].

3 The AHELM Framework

AHELM studies audio-language models that process interleaved audio and text as prompts to generate text completions. The evaluation process of AHELM comprises four primary components: aspect, scenario, adaptation, and metric (see Figure˜2).

An aspect refers to a particular evaluative dimension that aids in assessing overall performance. In AHELM, the aspects considered include audio perception, knowledge, reasoning, emotion detection, bias, fairness, multilinguality, robustness, toxicity, and safety (see  Section˜3.1 for details). These aspects are evaluated by calculating metrics across various scenarios.

A scenario denotes a use case for an ALM, characterized by a task (such as transcription, captioning, identifying emotion) and a usage category, which may include domain, language, or theme. For instance, a scenario like “audio question answering about emotions” involves the task of responding with the correct emotion in an audio clip after being asked. Our study encompasses a diverse array of scenarios, with tasks ranging from audio question answering to captioning, and usage categories that include multiple languages, subjects, and audio types. Scenarios employed in AHELM are listed in Table˜2.

A scenario consists of instances—defined as pairs of prompts and references—that can be used to evaluate model performance across one or more scenarios. A dataset can support multiple scenarios. For example, while FLEURS[9] is often used to assess audio perception, we can also assess fairness by detecting differences in the performance of the models given speech from different sexes. In some contexts, a dataset may be synonymous with a scenario, particularly in model evaluation. For example, we might refer to “Air-Bench (Foundation/Music)” as a scenario, implying that the music subset within the Air-Bench[44] (Foundation) evaluates audio question answering within the music domain. AHELM compiles a total of 14 existing datasets and adds 2 new datasets (refer to Table˜2).

An adaptation is a specific procedure for invoking a model. Adaptation strategies include zero-shot prompting, $k$-shot prompting, and chain-of-thought prompting. In this study, we exclusively employ zero-shot prompting, as it is the most prevalent strategy used by the general public.

A metric quantifies the performance of an ALM within a scenario. Examples of metrics include word error rates or scoring by either a human or a model on a scale from 1 to 5.

3.1 Aspects & Scenarios

AHELM evaluates ALMs on 10 technological and societal aspects that are critical for the deployment of safe and reliable ALMs. For each aspect, we identify scenarios that mainly evaluate it according to our definitions (see Table˜1). We aim to minimize overlaps in the scenario testing and choose the more popular or appropriate scenario when confronted with duplicates. For example, we use LibriSpeech only and forgo CSR-I (WSJ0) and Common Voice when testing for ASR capabilities (under Audio Perception). We create two new scenarios: CoRe-Bench and PARADE to appropriately measure complex, long audio reasoning (see “reasoning” paragraph and Appendix˜D) and ALM bias, respectively (see “bias” paragraph and Appendix˜E). The scenarios are listed in Table˜2 and we present detailed audio sampling rates of each scenario in Appendix˜B.

Audio perception refers to the capability of extracting meaningful information from audio signals. This ability can be assessed through various tasks, such as automatic speech recognition (ASR) and audio question answering (AQA). In ASR, audio language models (ALMs) are employed to convert spoken language into text, effectively transcribing audio inputs. On the other hand, AQA involves ALMs being challenged to answer questions that are based on audio inputs, thereby demonstrating their understanding and processing of auditory information.

Similar to LMs and Vision Language Models, ALMs are equipped with knowledge and reasoning capabilities. Knowledge refers to the model’s ability to recall facts or information embedded within its training data. This capability can be evaluated by posing questions that require the model to identify or recognize elements not explicitly present in the input audio.

Reasoning, conversely, involves the model’s ability to perform a series of logical inferences to deduce an answer. This is assessed by presenting questions whose answers are not directly stated in the inputs but can be inferred through a series of logical connections between speech, text, and sounds (e.g., imitation of the calls of animals). While existing benchmarks often emphasize surface cues or direct retrieval from text, they rarely challenge models to reason over dynamic, audio-grounded conversations [44]. To evaluate this capacity, we propose CoRe-Bench, a new benchmark for long conversational audio reasoning through carefully constructed, multi-turn dialogues paired with questions. Our goal is to minimize the need for cultural or factual knowledge (e.g., specific celebrities or media) and instead focus on personal attributes, such as genre preferences or demographics. This ensures accessibility across diverse populations and fairer evaluation of reasoning.

CoRe-Bench’s data construction process involves four stages: (1) generation of conversational scenarios based on demographic and relational parameters; (2) transcript creation using LMs; (3) answerability validation via automatic checking; and (4) audio synthesis using text-to-speech. All conversations center around questions probing personal preferences (e.g., “What is the favorite music genre of the first speaker?”). We also include adversarial examples with irrelevant questions that cannot be answered from the conversation to make it more challenging.

The resulting dataset consists of diverse, demographically grounded, and audio-based multi-turn conversations paired with questions and answers. It enables fine-grained evaluation of a model’s ability to reason over realistic audio dialogues. We present more details on construction steps, prompt design, validation criteria, data statistics, and data analyses in the Appendix D.

Emotion Detection is the ability to detect the user’s conscious mental state deriving from his mood, circumstances, or relationships with others. Sounds as expressed through speech or music is used by humans to express their feelings and it is important for ALMs to discern and understand them.

Bias in the context of Language Audio Models (ALMs) pertains to the model’s capacity to prevent forming inappropriate or unwarranted associations between its inputs and outputs. In ALMs, the audio input introduces an additional layer where such spurious correlations might arise, potentially leading to undesirable outcomes. For instance, the model might infer the speaker’s gender from their voice and subsequently generate outputs that reinforce gender stereotypes. To measure this, we introduce a novel dataset, PARADE, in this paper that presents an audio clip and asks for the most likely role of the speaker. The options in the question are contrasting roles that reflect either the occupation (e.g., doctor vs nurse) or the social status (e.g., rich vs poor) and the speech content is designed to be equally likely spoken by both roles (e.g., “Where is your pain?”). The gender of the voice is used as a confounding variable. PARADE contains a total of 938 examples spanning 20 occupation pairs and 5 status pairs. Every instance is synthetically verbalized by both male and female voices. We describe the dataset, including its construction, in detail in Appendix˜E.

Fairness pertains to two main concepts in AHELM: counterfactual fairness and performance disparity. Counterfactual fairness is concerned with ensuring that the model’s responses remain consistent when a non-essential or spurious attribute of the input is altered. For example, the word error rate should remain consistent regardless of whether the ALM is transcribing the same speech content spoken by a Latino or by an Asian. Performance disparity, on the other hand, refers to the model’s ability to perform uniformly across various subsets of the data, where each subset is defined by a particular attribute. For instance, when evaluating the model’s transcription accuracy across age groups, the model should achieve similar levels of accuracy whether the speakers are teenagers or seniors.

Multilinguality is the ability to execute tasks effectively even when the language of the instructions or the language of the output is altered. It enhances the ALMs’ versatility and applicability in diverse linguistic contexts and broadens their usability across different regions and cultures.

Robustness refers to the model’s ability to consistently generate accurate and desired outputs despite variations or disturbances in the input audio and/or text. These perturbations might include typographical errors in the text or environmental noise that affects the clarity of the audio input. The ideal ALM should be impervious to these perturbations.

Toxicity refers to the model’s capability to detect and reject offensive or harmful content, including hate speech, violent language, abusive remarks, and similar expressions. This capability is crucial for maintaining a safe and respectful environment in applications such as speech recognition systems or voice-activated assistants.

Safety involves ensuring that the model does not generate responses that could potentially harm humans. This is particularly important as audio is another vector of attack that can induce the model to generate responses that are either illegal or results in undesirable outcomes for the users.

3.2 Metrics

We implement automated metrics so that evaluations can be fast, consistent, and cheap to execute. For ASR tasks, we apply common metrics such as the word error rate (WER). For translation tasks, bilingual evaluation understudy (BLEU) score is used. For scenarios that consist of multiple-choice questions, the accuracy is used as the metric. To evaluate performance disparities in fairness, we perform two tests to determine if the difference across the groups is statistically significant:

1. we apply the $t$-test on the difference between the mean of the two groups.
2. we compute the difference in accuracies between paired samples and apply the paired samples $t$-test. Please see Appendix˜G for mathematical details.

For open-ended tasks such as captioning, we deploy an LM (i.e., GPT-4o) to evaluate whether the ALM’s output aligns with the reference text is used in order to provide consistent, cheap, and fast evaluation. While an ALM can be deployed as a judge, we reason that using an LM is cheaper and avoids the contradictory situation of having an ALM evaluate itself—which may bias the scores. We manually score 197 instances and find that the LM judge has an exact agreement rate of 50.8%, an $\pm$1 agreement rate of 83.8%, and a weighted kappa agreement of 83.3%, validating its use (see Section˜F.3).

Details of our LM judge, including its prompts and an analysis of its alignment with human scores, are described in Appendix˜F. GPT-4o is used as a judge for AudioCaps, Air-Bench Chat (reasoning subsets), and Air-Bench Chat (knowledge subsets).

Aggregation is performed at several levels. For each model and scenario, we average the main metrics (i.e., accuracy or word error rate) across all the instances to produce a summary score for that model on the scenario. We then use this to calculate the mean win rate—defined as the probability that the model outperforms another model selected uniformly at random for a given metric in a head-to-head comparison—for the model on that scenario. To produce the overall leaderboard, we compute the mean win rate for all the scenarios that covers that aspect.

4 Experiments

ALMs.

We consider only popular, state-of-the-art models in our evaluation to ensure meaningful and effective comparisons. This results selecting the Qwen family of models for open-weight models and Gemini and OpenAI models for closed-API models. We evaluate models from the same family to investigate how performance changes between model generations in a fair and controlled environment. In all, we assess a total of $14$ ALMs developed by $3$ different organizations (see Table˜3).

To guarantee equitable and reliable comparisons among ALMs, we standardize the inference parameters by setting the model temperature to 0 and the maximum number of output tokens to 200. All models are given the same zero-shot prompts and only one try per instance.

Baseline ASR and LM systems

In addition to testing ALMs, we benchmark LM-based systems consisting of an dedicated ASR module (either Whisper-1, GPT-4o Transcribe, or GPT-4o-mini Transcribe) that transcribes the input audio to text and an LM (i.e., GPT-4o) that has access to the transcribed text in addition to the input text prompt. These systems serve two purposes: Firstly, they allow us to gauge when and by how much can ALMs outperform simple engineered systems, if at all. Secondly, they provide useful information about the scenarios; for example, by checking how they perform on MELD—which probes the models to classify the emotions after listening to an audio clip—we can understand whether the emotional cues are provided by the content of the speech (validated if the baseline systems perform well) or from more subtle audio cues such as the speech inflection (validated if they perform poorly). We show the flow of data through the system and details of how we incorporate the transcribed text from the ASR into the LM prompt in Appendix˜C.

We randomly sample up to 1,000 instances per scenario for evaluation. To fully evaluate on AHELM, each model processes 39,538 instances, which consists of 5,728,718 characters of input text and 41,228 audio files in total. The generated output varies in length depending on the model and decoding parameters, as well as instructions embedded in the prompt. For context, Qwen2.5-Omni (7B) generated a total of 3,823,092 characters in its completions across all the scenarios. We conducted our experiments between February 16, 2025 and June 1, 2025.

5 Results and Analysis

We summarize the experimental results in this section. Due to page constraints, we relegate additional summaries to Appendix˜I. Visual representations of the aspect and scenario scores are shown in Figure˜A20 and Figure˜A19, respectively. Full result tables are archived in Appendix˜H. We encourage our readers to visit our benchmark website at https://crfm.stanford.edu/helm/audio/v1.0.0/, where we display the prompts, predictions, and scores for every model and instance.

There is no single model that excels across all scenarios. Among the ALMs, Gemini 2.5 Pro (05-06 Preview) is the overall best, scoring a mean win rate (MWR) of 0.803. It ranks top in 5 out of the 10 aspects with leaderboards: audio perception, reasoning, emotion detection, multilinguality, and robustness. 2. 2.

Open-weight models are generally weaker in instruction following, which in turn leads to degraded performance. For example, when prompted to “respond with only the transcript text and nothing else”, Qwen2-Audio Instruct instead outputs “The speech is in English, saying [correct transcript]”. Likewise, when prompted to output only one word that corresponds to the emotion, Qwen2.5-Omni will output the word followed by a string of explanations. We see remarkably better instruction following on the Qwen2.5-Omni than Qwen2-Audio Instruct, indicating that open-weight models are improving. 3. 3.

Dedicated ASR systems are more robust. While Gemini 2.5 Pro is the model most robust to environmental noise (WER of 0.039 on Robust Speech Bench), the dedicated ASR models (our baseline systems) are significantly more robust than most ALMs, ranking 2nd, 3rd, and 5th among all the models in the robustness aspect (see Table˜A19). The better performances of the baseline systems might be due to the specialized architecture and engineering optimizations used. 4. 4.

Baseline models reveal that there is a lot of information in the speech in the emotion detection scenarios. Gemini 2.5 Pro (05-06 Preview) scores the best on emotion detection (MWR: 0.781) while GPT-4o Audio (Preview 2024-12-17), Qwen2.5-Omni (7B), Gemini 1.5 Pro (002) and GPT-4o Transcribe + GPT-4o (2024-11-20) are tied for the second spot (see Table˜A11). Interestingly, the baseline systems are ranked 2nd to 4th, implying that there are already plenty of information in the speech content (in contrast to speech inflection or other audio cues) in these scenarios.

A closer look at the emotion detection scenarios indicate that while the baseline systems perform the best on MELD, they are ranked in the lower half in MUStARD. Given that the ASR models do not describe prosody or identify the speakers, we postulate that the MELD is a simpler benchmark consisting of sentences by a single speaker where the emotions can be inferred from the speech content. In contrast, detecting sarcasm in MUStARD requires understanding speech prosody and interactions between individuals. A manual inspection of the dataset confirms our suspicions. 5. 5.

While the model performances on toxicity detection (MuToX) are mixed, they all perform better in some languages than others. (Tables˜A20, A21 and A22) GPT-4o mini Audio did the best overall (mean accuracy of 87.4%), followed by the full-fledged GPT-4o Audio models (0.859 and 0.858 for Preview 2024-10-01 and Preview 2024-12-17, respectively). The baseline systems are in the middle of the pack (e.g., 8th of 17 for GPT-4o Transcribe + GPT-4o).

Looking at the mean MuToX scores by languages, we find it surprising that the models perform the best on French (EM: 0.956) and Indonesian (EM: 0.953) and perform the worst on Vietnamese (EM: 0.592) and English (EM: 0.579). Given that the baseline systems also display similar patterns, we hypothesize that the English subset is more difficult and/or is better curated. It may also be the case that the standard for toxicity may differ across the cultures and languages. 6. 6.

Current ALMs are generally robust to the speaker’s gender on ASR. Looking at the results for fairness (Tables˜A13 and A14), we observe that, in most cases, the models do not display statistically significant difference in performance when encountering speech by different sexes; In FLEURS, the paired-samples $t$-test detects a significant preference for females on Gemini 2.5 Pro (05-06) ($p$=0.02) and Qwen2.5-Omni ($p=$0.02) while the independent $t$-test detects a preference for females on Qwen 2.5 Omni ($p=$0.01) and on Qwen 2 Audio Instruct ($p=$0.03). LibreSpeech reveals that the Gemini family of models seems to have a lower WER when the speaker is a male ($p=$0.06 for Gemini 2.0 Flash, $p=$0.06 for Gemini 2.0 Flash (Experimental), and $p=$0.03 for Gemini 2.0 Flash Lite, $p=$0.00 for Gemini 2.5 Flash (05-20 preview)). This is not observed in Gemini 1.5. It also shows that GPT-4o-mini Transcribe works better when the speaker is male ($p=$0.01) even though GPT-4o Transcribe doesn’t exhibit statistically significant ASR bias when conditioned on the sex.

6 Discussion and Conclusion

6.1 Limitations

In this paper, we identify 10 aspects that we believe are important to the development and adoption of ALMs. While we identify missing datasets for some of the aspects (e.g., bias) and attempt to remedy it by introducing new ones (e.g., PARADE), it is possible that we have missed out other important aspects. Based on our analysis of the baseline systems’ performance on the scenarios, we highlight that some scenarios (e.g., MELD) may need improvements to better assess ALMs’ ability to extract information from non-speech content (e.g., intonation). As with all benchmarks, our results are technical objects that have to be contextualized to be useful. Further work to understand the nuances of the scores and correlate them to real-world impact is currently lacking and is left as future work.

6.2 Broader Impact

AHELM evaluates ALMs using a standardized set of prompts, scenarios, and metrics, enabling researchers, model developers, and deployment decision-makers to better understand the models’ strengths and weaknesses. Our introduction of baseline systems allows users to consider the possible alternative of using simpler ASR and LM system to accomplish their tasks. We also hope that these baseline systems can spur innovations, such as incorporating some ASR specific designs into ALM architectures to make them more robust to noises. As we have shown, ASR systems are strong at reliably transcribing human speech. It may be possible to integrate model components from ASRs into ALMs so as to enhance their performance, especially in speech recognition tasks. We leave this as a potential future directions Lastly, we hope that our method for generating synthetic dataset such as CoRe-Bench and PARADE can be adapted to create more benchmarks and training data. We hope that this will spur the community to develop newer and better datasets—both for training and assessing the capabilities of ALMs.

6.3 Conclusion

This paper introduces AHELM, a benchmark that evaluates ALMs across 10 important aspects, thereby enabling developers and users to quickly and fairly measure and compare model capabilities. AHELM introduces multiple innovations, such as the CoRe-Bench and PARADE scenarios and novel use of ASR+LM to identify weaknesses in evaluation datasets. AHELM will be a living benchmark where models and scenarios will be added over time as they emerge.

Acknowledgments

We thank Google and Hitachi for their support for the project. We also thank Microsoft Accelerate Foundation Models Research Program and the OpenAI Researcher Access Program for supporting our computing needs. The views and opinions expressed in this article are those of the authors only and do not necessarily represent the views and opinions of any other organization, any of their affiliates or employees acknowledged above.

Appendix A Aspect coverage

Appendix B Sampling Rates of Scenarios

Appendix C ASR+LM baseline system

Our baseline system consists of a dedicated ASR paired with a LM. The ASR model transcribe the input audio clips into text, transcribed_audio, which will be fed as part of the prompt into the LM.

See Figure˜A2 of an example of the input prompts. In our implementation, we try various combinations, using Whisper-1, GPT-4-transcribe, or GPT-4-mini-transcribe as the dedicated ASR model and GPT-4o as the LM.

Appendix D CoRe-Bench: Audio Conversational Reasoning Benchmark

While ALMs have found uses in some commercial software as voice assistants on mobile devices, they often converse with a single speaker and accept short and simple prompts. It is unclear if the ALMs can understand and reason through long, complex conversations involving multiple speakers—a necessary skill if they are to be deployed in more sophisticated situations such as to take minutes in an on-site meeting with multiple participants. To the best of our knowledge, there is no benchmark that assesses this capability comprehensively.

An instance in the ideal conversational reasoning benchmark will require ALMs to identify speakers and understand the context of the conversation and the information conveyed by each speaker before reasoning through the information given to derive the most probable answer. Within the data set, the instances should be diverse in terms of i) conversational content, ii) length of conversation, iii) voices (gender and emotions), iv) complexity (e.g., number of people). Furthermore, it should be cheap and scalable.

Creating such a benchmark is non-trivial. One possible approach is to hire humans to write and record play scripts and come up with plausible questions and answers. While this results in customizable, high quality data, it is expensive to produce and difficult to scale. Another possible approach is to scrape and extract audio conversations from podcasts or videos on the internet and create question and answer pairs from them. It avoids the need to create conversations but introduces the inherently difficult task of generating relevant questions whose answers can be obtained from the pre-defined speeches. The questions generated through this method are often a rehash of the conversation and as such, the answers can obtained without much difficulty.

Here, we introduce an fully automatic pipeline to create synthetically generated conversations, questions, and answers cheaply and quickly using state-of-the-art large language models and steerable text-to-speech models. Our resulting benchmark, CoRe-Bench, contains 2290 question-answer pairs grounded in 2082 unique multi-turn audio clips, amounting to over 48 hours of dialogue. To ensure broad coverage and variability, the conversations span over 3,800 distinct scenarios across speaker age groups, relationships, and culturally appropriate topics. The dialogues range in length from 24.5 to 230.2 seconds, involve 2 to 5 speakers, and are voiced using 11 distinct speakers (7 male, 4 female) with varied affective and vocal profiles. Each question is designed to require inference based on the full context of the conversation, rather than surface-level retrieval.

In the following section, we detail the construction pipeline, question design, validation procedure, audio generation, and dataset statistics that underpin CoRe-Bench.

D.1 Dataset Construction

Figure˜A3 shows an overview of the data construction process, which consists of 4 major steps: scenario generation, transcript generation, question-answer verification, and audio generation.

D.1.1 Scenario generation

In the scenario generation step, structured inputs, such as the age of the speakers and the generic relationships between them are fed as part of a prompt to an language model that instructs it to generate random conversational scenarios, which provide context for generating conversations111A conversational scenario is different from the AHELM scenario introduced in Section 3. Each scenario consists of the relationship between the speakers, the verb, the topic of discussion, the environment that they are in, and the mood of the conversation. Each call to the language model requests for 50 unique scenarios. Multiple calls are made and the responses are then compiled and deduplicated. In all, we generated 3,883 unique scenarios from GPT-4o, whose temperature is set to 0.7 in this step to induce diversity. See Figure˜A4 for the prompt.

D.1.2 Transcript generation

This step uses an LM to generate conversational transcripts. The input prompt to generate the transcript contains a seed question, two possible answers, details about the speakers such as their names, age groups, and region, conversation details such as the desired number of dialogues, and the scenario.

We maintain a predefined set of 20 seed questions focused on personal preferences and attributes of speakers that are formatted as "What is the favorite X of the first speaker?", where X represents various subjects such as book genres, music genres, or sports, among others. The list was generated by GPT-4o but manually curated by the authors. We also maintain a pre-defined list of regions that are also generated by GPT-4o but manually curated by the authors.

For each question and region, the system generates a bank of possible answers using GPT-4o with the prompt shown in Figure˜A5. The LM is queried 20 times per question and repeated entries are deduplicated to ensure diversity in the answer bank. The end of sentence phrase “…always return the English name” is necessary because LMs may sometimes misinterpret the instruction and produce nouns in the regional language (e.g., “aglio” instead of “garlic”). For each region, we also keep a list of possible names of the speakers for the region. The names are generated in a separate LM step with the prompt shown in Figure˜A6. Again, the LM is queried multiple times and the responses are then compiled and deduplicated.

Finally, a random set of parameters consisting of a scenario, a region, a seed question, two possible answers (but only one is valid), the number of speakers, the number of dialogues, and a list of names of the speakers are generated and included as part of a prompt that instructs the LM to generate a conversation (see Figure˜A7) and an associated answer. The strategy of forcing to use the two possible answers to the seed question in the conversation generation is an result of experimentation. Prior iterations without this strategy generated conversations whose answers that can easily be guessed. For example, one can easily answer “what is the favorite flower of [speaker]?” by doing a vocabulary search over the names of flowers in the conversation and finding only an unique result. With our two possible answers strategy, a confounding answer will be generated, which makes it much more difficult to guess the answer.

D.1.3 Question-and-answer verification

We generate the transcripts using either GPT-4o or Gemini-2.5 Flash Preview (04-17) (selected at random) and use the other LM (i.e., Gemini is used as validator if the transcript is generated by GPT-4o) to attempt to answer the question from the transcript. To do this, we mask the names of the speakers in the transcript to simulate that fact that these are not known in an audio setting and feed both the transcript and the question to the validator (input prompt is shown in Figure˜A8). The output of the validator is then matched against the answer generated by the transcript generator using either GPT-4o-mini (prompt shown in Figure˜A9). This entire process makes sure that the question is answerable from the conversation. The use of the different LMs for the generator and the validator minimizes possible model bias, which may exist as both the LMs and ALMs may have been trained on the similar data within the company. If the validation fails, the conversation and answer are generated again. We attempt 3 times before giving up.

D.1.4 Audio conversation generation

We convert the transcripts into audio conversations using synthetic text-to-speech engines. In particular, gpt-4o-mini-tts is used as it allows users to steer the accent, emotional, intonation, speech speed, and tone to generate natural sounding spoken text.

We assign each speaker to the model’s set of 7 male voices and 4 female voices based on their sex and generate each turn of the dialogue separately before combining them together using the pydub library. The input prompt to the TTS, as seen in Figure˜A10, contains the speech patterns such as voice (e.g., “humorous and imaginative”), tone (e.g., “joking and creative”), accent (“Portuguese (European)”), and features (“slight lilt”). These speech patterns and the pauses between the turns are generated by the LM in the transcript generation stage (see Figure˜A7).

D.2 Audio statistics

We create 2082 audio conversations ranging between 24.5s and 230.2s. The average length of the audio is 1m 23.2s and the standard deviation is 26.5s. In total, we produce over 48 hours worth of audio artifacts. The statistics are visualized in Figure˜A11.

The generation of the entire dataset takes less than an hour (including rate limits on API calls) when executed on a 64 cores (128 threads) machine, demonstrating the scalability of our approach.

D.3 Augmentation with irrelevant questions

We replace the original questions with random questions to create instances where the question cannot be answered by the conversation, which allows us to test the models on hallucination. The random questions are created by prompting LLMs to produce long and convoluted questions (e.g., “What is the theme of the holiday celebrated in the enchanted village where villagers dress up as animals and exchange handmade crafts every winter solstice”) pertaining to a question category (e.g., holiday). We chose this method over shuffling the question and answer pairs as it minimizes the chances that the question can actually be answered by the original conversation.

The final number of instances is 2290 , of which 208 ($\sim$ 9.1%) are unanswerable. The dataset is released on Huggingface at https://huggingface.co/datasets/stanford-crfm/CoReBench_v1.

D.4 Analysis of CoRe-Bench

We perform simple analyses of the performances of the model on CoRe-Bench beyond what is presented in the rest of the paper here.

D.4.1 Accuracy of the models improves marginally with number of dialogues

In Figure˜A13, we plot the accuracy of the models against the number of dialogue turns in the conversations. As can be seen, the mean accuracy of the models improves only marginally with the number of dialogues.

D.4.2 Accuracy is independent of number of speakers.

D.4.3 Accuracy differs by question subject.

From Figure˜A15, we observe that models perform badly on "what is the name of the first/second/… speaker?" problems, indicating that they actually are quite bad in terms of either reasoning names or at the cocktail party problem.

D.4.4 OpenAI models are most likely to falsely tag the questions as ‘unanswerable’.

We create ‘unaswerable’ instances to assess if the models can follow text instructions and relate the text to the audio. We quantify this by treating ‘unanswerable’ instances as the positive class and computing the F1 scores. As can be seen from Table˜A3, the models is in general still a problem. OpenAI models have high recall but low precision (i.e., they just answer “unanswerable” as much as possible), leading to low F1 scores. Gemini models are a lot better, but can still improve.

Appendix E PARADE: A benchmark for audio bias

While we identify the need for models to avoid biased response conditioned on the speaker, we are unable to identify a dataset that evaluates this aspect. To address this gap, we construct a dataset, PARADE, that probes whether the model produce stereotyped output. PARADE is inspired by PAIRS [15], which examines how vision-language models respond to different bias cues (i.e., occupation, status, and crime) in images and questions.

PARADE consists of audio clips and multiple-choice questions that ask for the role of the speaker. The options in the questions reflect contrasting roles—drawn from either occupations or statuses—that could have made the speech. Figure˜A16 shows an instance in the dataset. Since the gender of the speaker is irrelevant, it serves as a confounding variable. We say that the ALM is unbiased if it is unsure of its answers or refuses to answer our questions and claim that it makes unwanted associations otherwise.

In this section, we document the construction of and the statistics for the dataset.

E.1 Dataset construction

The dataset is constructed in three stages:

We obtain a list of contrasting roles. 2. 2.

We generate transcripts of utterances that could be spoken by by both roles using an LM. 3. 3.

We generate audio speech using text-to-speech engines.

E.1.1 Obtaining a list of contrasting roles

We use the list of roles from PAIRS (replicated in Table˜A4) to seed the generation of speech content. In the current iteration of PARADE, we do not explore new roles or categories.

E.1.2 Generation of transcript

We prompt GPT-4o to generate utterances that could be spoken by both roles. The input prompt is shown in Figure˜A17. After generating the list of sentences, we manually filter out those that are incomplete and incoherent.

E.1.3 Generation of synthetic audio

For the synthetic test audios, we employ OpenAI’s state-of-the-art text-to-speech (TTS) model222https://platform.openai.com/docs/guides/text-to-speech.. To ensure voice diversity, we generate audio using both female and male voices. Specifically, we synthesize female speech with the nova voice and male speech with the onyx voice provided by OpenAI’s TTS system.

E.2 Summary statistics

In total, we collect 738 audio samples (369 transcripts $\times$ 2 voices) that assesses occupational bias and 200 (100 transcripts $\times$ 2 voices) that assess social status bias. We present three transcript samples each from three occupation pairs and three status pairs in Table A5. The transcripts and audio data are hosted at https://huggingface.co/datasets/UCSC-VLAA/PARADE_audio.

Appendix F GPT-4o as a judge for audio scenarios

Multimodal language models have been used as judges has been used for various scenarios. For example, [13] and [12] use LM to simulate human feedback for the purpose of evaluating LM output. [27] uses Prometheus-Vision [26] as a judge for benchmarks that take both images and text as input and produce freeform text as output.

Since the reference text are available for the scenarios in AHELM, we eschew the use of ALMs as evaluators and instead use use LMs to evaluate whether the ALM text output aligns with the ground-truths. In addition to being a cheaper method for evaluation, the use of LM avoids the contradictory situation of having an ALM to evaluate itself, which may bias the scores. LLM-as-a-judge is used for AudioCaps, Air-Bench Chat (reasoning subsets), and Air-Bench Chat (knowledge subsets).

F.1 Methodology

Given a reference answer $r$ and a model response $o$, we ask GPT-4o to evaluate $o$ against $r$ with the following rubric:

1. Score 1:

The response is completely inaccurate or unrelated to the ground truth. 2. Score 2:

The response contains significant inaccuracies or misinterpretations that distort the meaning of the ground truth. 3. Score 3:

The response is mostly accurate but includes minor errors, omissions, or ambiguities. 4. Score 4:

The response is accurate and aligns well with the ground truth, with only slight room for improvement. 5. Score 5:

The response is fully accurate and precisely matches the ground truth with no errors or misinterpretations.

The LM is asked to produce a single score with a single line explanation for every evaluation (see Figure˜A18).

F.2 Human Evaluation

We measure the goodness of the LM judge by manually rating samples and computing the LM’s alignment with the human scores. We obtain 197 random samples and have 4 human raters label them with the exact same rubric as presented to the LM. Each sample is rated by 1 rater only. We compute the exact agreement rate, the $\pm$1 agreement rate, and the Cohen’s $\kappa$, the last being a more appropriate metric for ordinal data [8].

F.3 Results

We find that GPT-4 critic has an exact agreement rate of 50.8%, a $\pm$1 agreement rate of 83.8% with respect to the human scores (see Table˜A6), and a Cohen’s $\kappa$ of 83.8% (see Table˜A7), demonstrating that LMs can provide consistent judgments that often align with human evaluators.

We also test four additional LMs—LLaMA-3.1-8B-Instruct, Qwen-2.5-32B, LLaMA-3.3-70B-Instruct, and Claude 4 Sonnet—to investigate the impact of using different LMs as judges (see Table˜A7). We find that GPT-4o produces the highest alignment with human rating, validating once again its use as the judge in our study.

We note that while using an LLM as a judge allows quick and cheap evaluation of open-ended responses, it may introduce subtle issues such as self-preference, consistency, position bias, or preference for longer output. While we have demonstrated that GPT-4o as a judge aligns best with human preferences, we have yet to explore how the use of different judges will impact the stability of the leaderboards. This is left as future work.

Appendix G Analysis of the fairness scenarios

Analysis of fairness scenarios generally into one of the following two types: independent groups and paired samples.

[Independent groups]

We create two subsets of benchmark instances, one comprising of males and the other comprising of females. Define the mean score of the ALMs on the male and female subsets to be $\mu_{\text{male}}$ and $\mu_{\text{male}}$ If the ALM performs the same between the two groups, we will expect that $\hat{\mu}_{\text{male}}=\hat{\mu}_{\text{female}}$. This can be tested using a 2-sided $t$-test:

[TABLE]

The $t$-stat can be computed as:

[TABLE]

where, $\bar{x}_{g}$ is the sample mean, $s^{2}_{g}$ is the sample variance, and $n^{2}_{g}$ is the number of members in group $g$.

This test is used in both the FLEURS (fairness) and LibriSpeech (fairness) scenarios. See Section˜H.6 for the analyses.

[Paired samples]

Paired samples occur when the same content is recited by at least one male and at least one female. Given the scores $c_{i}$ across all content, the paired difference $d_{i}$ can be defined as:

[TABLE]

Given the hypothesis:

[TABLE]

The paired-sample $t$-stat can be computed as:

[TABLE]

where $\bar{d}=\frac{1}{n_{d}}\sum_{i}d_{i}$ is the arithmetic mean of the sample differences and $s_{d}$ is the standard deviation of the sample differences.

This test is applied only on the paired samples in the FLEURS (fairness) scenario. See Table˜A13 for the analysis.

Appendix H Results

H.1 Audio Perception

H.2 Knowledge

H.3 Reasoning

H.4 Emotion Detection

H.4.1 Selected examples

H.5 Bias

H.6 Fairness

This section presents the result of our statistical analysis on the fairness scenarios.

H.7 Multilinguality

H.8 Robustness

H.9 Toxicity

Tables˜A20, A21 and A22 shows the overall results for toxicity detection. GPT-4o mini Audio did the best overall (mean accuracy of 87.4%), followed by the full-fledged GPT-4o Audio models (0.859 and 0.858 for Preview 2024-10-01 and Preview 2024-12-17, respectively). The baseline systems are in the middle of the pack (e.g., 8th of 17 for GPT-4o Transcribe + GPT-4o).

Looking at the breakdown by languages, we find it surprising that the models perform the best on French (mean EM: 0.956) and Indonesian (mean EM: 0.953) and perform the worst on Vietnamese and English. Given the fact that the baseline systems also perform well on French and Indonesian, among others, we hypothesize that the English subset contains more difficult instances and/or is better curated. It may also be the case that the standard for toxicity may differ across the cultures and languages.

H.10 Safety

Appendix I Additional Results

Here we present additional results in addition to those in the main paper.

The ‘transcribe + LM’ paradigm falls short in more ‘natural’ tasks. Comparing the dedicated ASR models, we observe that GPT-4o Transcribe and GPT-4o Mini Transcribe fail to transcribe properly when fed speech in more ‘natural’ settings. For example, in MUStARD, where the audio clips are extracted from sitcoms such as FRIENDS or Big Bang Theory and consists of alternating dialogue with potentially long pauses, the transcriptions by GPT-4o Transcribe and GPT-4o Mini Transcribe are often incomplete. In these cases, Whisper-1 is able to transcribe the entire dialogue but fails to identify the speakers. See Section˜H.4.1 for examples. On the other hand, we observe that GPT-4o Transcribe and GPT-4o Mini Transcribe are able to transcribe human sounds beyond speaking such as laughter (e.g., “haha”) or throat clearing (e.g., “ahem”) whereas Whisper-1 does not, leading to these models performing better on VocalSounds (see Table˜A8). 2. 7.

Gemini and baselines perform well on multilinguality but performances are skewed towards internet data distribution. The baseline systems and the Gemini models dominate the top half of the multilinguality leaderboard, with GPT-4o Transcribe + GPT-4o (2024-11-20) performing the best, followed by Gemini 1.5 Pro (002) and then Gemini 2.5 Pro (05-06 preview). This suggests that chaining specialized capabilities can deliver good performances.

Looking at CoVost-2 (Table˜A16), we observe that all the models perform better on Spanish-to-English than on Chinese-to-English, reflecting a possible skew in the distribution toward Latin languages in many of the training datasets. This is also observed in the FLEURS (multilingual) scenario (Table˜A17), where the models perform better on English and Finnish than on Hebrew, Bengali, and Thai. 3. 8.

Open-weight models can compete head-to-head with the best closed-API models on audio knowledge. From Table˜A9, we see that Qwen2-Audio Instruct takes the lead in audio knowledge, followed by Gemini 2.5 Pro (05-06 Preview) and then Gemini 2.0 Flash. The baseline systems score worst in this aspect, indicating that the scenarios cannot be easily solved without access to the non-speech audio content (e.g., music). 4. 9.

OpenAI’s models are better at defending against jailbreak attacks. When looking at the safety aspect, we see that OpenAI models are robust to the voice jailbreak attack. It may be possible that this vulnerability has specifically been patched by OpenAI since the original paper [38] demonstrated successful attacks against GPT-4o. Qwen 2.5 Omni and Gemini 2.5 Pro refused only 51.1% and 53.3% of the time despite outperforming the OpenAI models on many other aspects.

NeurIPS Paper Checklist

Claims 2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? 3. Answer: [Yes] 4. Justification: We present our thesis in Section 1. We explain our benchmark in Section 3 and describe the experiments in Section 4 and report results in Section 5. 5. Guidelines:

• •

The answer NA means that the abstract and introduction do not include the claims made in the paper.

• •

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

• •

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

• •

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 6. 2.

Limitations 7. Question: Does the paper discuss the limitations of the work performed by the authors? 8. Answer: [Yes] 9. Justification: We discuss limitations in Section 6.1 10. Guidelines:

• •

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

• •

The authors are encouraged to create a separate "Limitations" section in their paper.

• •

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

• •

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

• •

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

• •

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

• •

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

• •

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 11. 3.

Theory assumptions and proofs 12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 13. Answer: [Yes] 14. Justification: See Section 6.2. 15. Guidelines:

• •

The answer NA means that the paper does not include theoretical results.

• •

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

• •

All assumptions should be clearly stated or referenced in the statement of any theorems.

• •

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

• •

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

• •

Theorems and Lemmas that the proof relies upon should be properly referenced. 16. 4.

Experimental result reproducibility 17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? 18. Answer: [Yes] 19. Justification: All of our results are reproducible with open-sourced evaluation recipe, data, and framework at https://github.com/stanford-crfm/helm. 20. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

• •

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

• •

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

• •

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

1. (a)

If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. 2. (b)

If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. 3. (c)

If the contribution is a new model (e.g., a language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 4. (d)

We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 21. 5.

Open access to data and code 22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? 23. Answer: [Yes] 24. Justification: We provide the exact code to reproduce our results at https://github.com/stanford-crfm/helm and the new datasets at https://huggingface.co/datasets/UCSC-VLAA/PARADE_audio and https://huggingface.co/datasets/stanford-crfm/CoReBench_v1. 25. Guidelines:

• •

The answer NA means that paper does not include experiments requiring code.

• •

Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

• •

While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

• •

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

• •

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

• •

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

• •

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

• •

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 26. 6.

Experimental setting/details 27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? 28. Answer: [Yes] 29. Justification: See Section 4. 30. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

• •

The full details can be provided either with the code, in appendix, or as supplemental material. 31. 7.

Experiment statistical significance 32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? 33. Answer: [Yes] 34. Justification: We report the $p$-value for the fairness aspect on LibriSpeech (fairness) in the Appendix E.6. But we do not compute error bars for other scenarios. Repeating the experiments with multiple runs is prohibitively expensive and negates the benefits of sampling. Given the large number of instances (1,000 exmaples) and usage categories for each aspect, we believe that we still obtain significant measurements that will reflect the true model performances. 35. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

• •

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

• •

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

• •

The assumptions made should be given (e.g., Normally distributed errors).

• •

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

• •

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

• •

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

• •

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 36. 8.

Experiments compute resources 37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? 38. Answer: [Yes] 39. Justification: We state the number of input and output text tokens used in our evaluation in Section 4. 40. Guidelines:

• •

The answer NA means that the paper does not include experiments.

• •

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

• •

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

• •

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 41. 9.

Code of ethics 42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 43. Answer: [Yes] 44. Justification: We followed the NeurIPS Code of Ethics and discussed our limitations in Section 6.1. 45. Guidelines:

• •

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

• •

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

• •

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 46. 10.

Broader impacts 47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 48. Answer: [Yes] 49. Justification: See Section 6.1. 50. Guidelines:

• •

The answer NA means that there is no societal impact of the work performed.

• •

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

• •

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

• •

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

• •

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

• •

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 51. 11.

Safeguards 52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 53. Answer: [Yes] 54. Justification: In this paper, we release PARADE dataset. Before transforming transcripts to audio, we performed human scrutiny of the audio transcripts to make sure that there is no improper or toxic content in the metadata. 55. Guidelines:

• •

The answer NA means that the paper poses no such risks.

• •

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

• •

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

• •

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 56. 12.

Licenses for existing assets 57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? 58. Answer: [Yes] 59. Justification: All the models used in the experiments are hosted either by HuggingFace or the respective makers. We cite all the datasets and models used in our work. We own the license to the HELM codebase. 60. Guidelines:

• •

The answer NA means that the paper does not use existing assets.

• •

The authors should cite the original paper that produced the code package or dataset.

• •

The authors should state which version of the asset is used and, if possible, include a URL.

• •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

• •

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

• •

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

• •

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

• •

If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 61. 13.

New assets 62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 63. Answer: [Yes] 64. Justification: **We provide executable codes in our codebase: https://github.com/stanford-crfm/helm.

PARADE is available at https://huggingface.co/datasets/UCSC-VLAA/PARADE_audio.

CoRe-Bench is available at https://huggingface.co/datasets/stanford-crfm/CoReBench_v1** 65. Guidelines:

• •

The answer NA means that the paper does not release new assets.

• •

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

• •

The paper should discuss whether and how consent was obtained from people whose asset is used.

• •

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 66. 14.

Crowdsourcing and research with human subjects 67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? 68. Answer: [N/A] 69. Justification: This paper does not contain crowdsourcing experiments. 70. Guidelines:

• •

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

• •

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

• •

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 71. 15.

Institutional review board (IRB) approvals or equivalent for research with human subjects 72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? 73. Answer: [N/A] 74. Justification: This paper does not include participants study that requires IRB approvals. 75. Guidelines:

• •

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

• •

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

• •

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

• •

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 76. 16.

Declaration of LLM usage 77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required. 78. Answer: [Yes] 79. Justification: As detailed in the Appendix B, we leverage OpenAI’s GPT-4o to create audio transcripts for the curation of PARADE benchmark. We further conduct human inspection of these scripts to filter out inappropriate and inaccurate content. 80. Guidelines:

• •

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

• •

Please refer to our LLM policy (https://neurips.cc/Conferences/2025/LLM) for what should or should not be described.