A Novel Automatic Framework for Speaker Drift Detection in Synthesized Speech

TL;DR

This paper introduces an automatic framework for detecting speaker drift in synthesized speech, combining cosine similarity analysis with large language models to improve coherence in long-form TTS outputs.

Contribution

It presents the first automated method for speaker drift detection using geometric analysis and LLM reasoning, supported by a new synthetic benchmark with human annotations.

Findings

Cosine similarity effectively captures speaker consistency within utterances.

LLMs can assess speaker drift based on structured speech representations.

The framework outperforms baseline methods in detecting speaker drift.

Abstract

Recent diffusion-based text-to-speech (TTS) models achieve high naturalness and expressiveness, yet often suffer from speaker drift, a subtle, gradual shift in perceived speaker identity within a single utterance. This underexplored phenomenon undermines the coherence of synthetic speech, especially in long-form or interactive settings. We introduce the first automatic framework for detecting speaker drift by formulating it as a binary classification task over utterance-level speaker consistency. Our method computes cosine similarity across overlapping segments of synthesized speech and prompts large language models (LLMs) with structured representations to assess drift. We provide theoretical guarantees for cosine-based drift detection and demonstrate that speaker embeddings exhibit meaningful geometric clustering on the unit sphere. To support evaluation, we construct a high-quality synthetic benchmark with human-validated speaker drift annotations. Experiments with multiple state-of-the-art LLMs confirm the viability of this embedding-to-reasoning pipeline. Our work establishes speaker drift as a standalone research problem and bridges geometric signal analysis with LLM-based perceptual reasoning in modern TTS.