MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions

TL;DR

MultiVox is a new benchmark designed to evaluate how well voice assistants understand and integrate multimodal cues, including speech paralinguistics and visual information, for more context-aware responses.

Contribution

It introduces the first comprehensive benchmark for assessing multimodal understanding in voice assistants, focusing on paralinguistic speech features and visual cues.

Findings

Current models underperform compared to humans in multimodal understanding.

MultiVox includes 1000 annotated dialogues with diverse speech and visual cues.

Evaluation reveals significant gaps in models' ability to generate contextually grounded responses.

Abstract

The rapid progress of Large Language Models (LLMs) has empowered omni models to act as voice assistants capable of understanding spoken dialogues. These models can process multimodal inputs beyond text, such as speech and visual data, enabling more context-aware interactions. However, current benchmarks fall short in comprehensively evaluating how well these models generate context-aware responses, particularly when it comes to implicitly understanding fine-grained speech characteristics, such as pitch, emotion, timbre, and volume or the environmental acoustic context such as background sounds. Additionally, they inadequately assess the ability of models to align paralinguistic cues with complementary visual signals to inform their responses. To address these gaps, we introduce MultiVox, the first omni voice assistant benchmark designed to evaluate the ability of voice assistants to integrate spoken and visual cues including paralinguistic speech features for truly multimodal understanding. Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features and a range of visual cues such as images and videos. Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.