From Signal to Turn: Interactional Friction in Modular Speech-to-Speech Pipelines

TL;DR

This paper investigates interactional friction in modular speech-to-speech AI systems, identifying key patterns of conversational breakdown and emphasizing the importance of infrastructure design for natural spoken AI.

Contribution

It introduces a system-level analysis of conversational issues in modular S2S-RAG pipelines, highlighting structural causes beyond simple latency or errors.

Findings

Identifies three patterns of conversational breakdown: Temporal Misalignment, Expressive Flattening, Repair Rigidity.

Shows that these issues are structural consequences of modular design, not just errors.

Suggests infrastructure-focused solutions for improving naturalness in spoken AI.

Abstract

While voice-based AI systems have achieved remarkable generative capabilities, their interactions often feel conversationally broken. This paper examines the interactional friction that emerges in modular Speech-to-Speech Retrieval-Augmented Generation (S2S-RAG) pipelines. By analyzing a representative production system, we move beyond simple latency metrics to identify three recurring patterns of conversational breakdown: (1) Temporal Misalignment, where system delays violate user expectations of conversational rhythm; (2) Expressive Flattening, where the loss of paralinguistic cues leads to literal, inappropriate responses; and (3) Repair Rigidity, where architectural gating prevents users from correcting errors in real-time. Through system-level analysis, we demonstrate that these friction points should not be understood as defects or failures, but as structural consequences of a modular design that prioritizes control over fluidity. We conclude that building natural spoken AI is an infrastructure design challenge, requiring a shift from optimizing isolated components to carefully choreographing the seams between them.