From Reactive to Proactive: Assessing the Proactivity of Voice Agents via ProVoice-Bench

TL;DR

This paper introduces ProVoice-Bench, a new evaluation framework for proactive voice agents, highlighting current models' limitations in proactive intervention, reasoning, and over-triggering through a comprehensive benchmark with novel tasks.

Contribution

The paper presents ProVoice-Bench, the first dedicated benchmark for proactive voice agents, including four novel tasks and a large dataset for evaluating multimodal LLMs.

Findings

State-of-the-art models show significant performance gaps in proactive tasks.

Models tend to over-trigger and lack reasoning capabilities.

ProVoice-Bench reveals limitations and guides future development of natural, context-aware agents.

Abstract

Recent advancements in LLM agents are gradually shifting from reactive, text-based paradigms toward proactive, multimodal interaction. However, existing benchmarks primarily focus on reactive responses, overlooking the complexities of proactive intervention and monitoring. To bridge this gap, we introduce ProVoice-Bench, the first evaluation framework specifically designed for proactive voice agents, featuring four novel tasks. By leveraging a multi-stage data synthesis pipeline, we curate 1,182 high-quality samples for rigorous testing. Our evaluation of state-of-the-art Multimodal LLMs reveals a significant performance gap, particularly regarding over-triggering and reasoning capabilities. These findings highlight the limitations of current models and offer a roadmap for developing more natural, context-aware proactive agents.