Defense Against Synthetic Speech: Real-Time Detection of RVC Voice Conversion Attacks

TL;DR

This paper presents a real-time detection system for AI-generated voice conversion attacks, using acoustic features and machine learning to distinguish authentic speech from RVC-generated deepfakes in streaming audio.

Contribution

It introduces a low-latency, segment-based detection method for RVC voice conversion, emphasizing realistic audio conditions and practical deployment considerations.

Findings

Short-window acoustic features effectively distinguish RVC speech.

Detection system performs reliably even in noisy environments.

Supports both segment-level and call-level classification for real-time use.

Abstract

Generative audio technologies now enable highly realistic voice cloning and real-time voice conversion, increasing the risk of impersonation, fraud, and misinformation in communication channels such as phone and video calls. This study investigates real-time detection of AI-generated speech produced using Retrieval-based Voice Conversion (RVC), evaluated on the DEEP-VOICE dataset, which includes authentic and voice-converted speech samples from multiple well-known speakers. To simulate realistic conditions, deepfake generation is applied to isolated vocal components, followed by the reintroduction of background ambiance to suppress trivial artifacts and emphasize conversion-specific cues. We frame detection as a streaming classification task by dividing audio into one-second segments, extracting time-frequency and cepstral features, and training supervised machine learning models to classify each segment as real or voice-converted. The proposed system enables low-latency inference, supporting both segment-level decisions and call-level aggregation. Experimental results show that short-window acoustic features can reliably capture discriminative patterns associated with RVC speech, even in noisy backgrounds. These findings demonstrate the feasibility of practical, real-time deepfake speech detection and underscore the importance of evaluating under realistic audio mixing conditions for robust deployment.