Speak or Stay Silent: Context-Aware Turn-Taking in Multi-Party Dialogue

TL;DR

This paper addresses the challenge of turn-taking in multi-party voice conversations, proposing a context-aware method that improves AI assistant behavior by explicitly training for appropriate speaking decisions.

Contribution

The paper introduces a new benchmark dataset and a supervised fine-tuning approach with reasoning traces to enable context-aware turn-taking in multi-party dialogue systems.

Findings

Large language models fail at zero-shot turn-taking in multi-party settings.

Supervised fine-tuning with reasoning traces significantly improves turn-taking accuracy.

Explicit training is necessary for effective context-aware turn-taking, as it is not an emergent capability.

Abstract

Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.