Still Between Us? Evaluating and Improving Voice Assistant Robustness to Third-Party Interruptions

TL;DR

This paper introduces TPI-Train and TPI-Bench, new datasets and evaluation tools to improve voice assistants' ability to handle third-party interruptions by emphasizing acoustic cues over semantic context.

Contribution

It presents novel datasets and evaluation frameworks that enhance speaker discrimination and interruption handling in spoken language models.

Findings

Dataset design reduces semantic shortcut learning.

Framework effectively measures interruption handling and speaker discrimination.

Code is publicly available at https://tpi-va.github.io

Abstract

While recent Spoken Language Models (SLMs) have been actively deployed in real-world scenarios, they lack the capability to discern Third-Party Interruptions (TPI) from the primary user's ongoing flow, leaving them vulnerable to contextual failures. To bridge this gap, we introduce TPI-Train, a dataset of 88K instances designed with speaker-aware hard negatives to enforce acoustic cue prioritization for interruption handling, and TPI-Bench, a comprehensive evaluation framework designed to rigorously measure the interruption-handling strategy and precise speaker discrimination in deceptive contexts. Experiments demonstrate that our dataset design mitigates semantic shortcut learning-a critical pitfall where models exploit semantic context while neglecting acoustic signals essential for discerning speaker changes. We believe our work establishes a foundational resource for overcoming text-dominated unimodal reliance in SLMs, paving the way for more robust multi-party spoken interaction. The code for the framework is publicly available at https://tpi-va.github.io