FreeTalk:A plug-and-play and black-box defense against speech synthesis attacks
Yuwen Pu, Zhou Feng, Chunyi Zhou, Jiahao Chen, Chunqiang Hu, Haibo Hu, Shouling Ji
TL;DR
This paper introduces FreeTalk, a lightweight, robust, plug-and-play black-box defense method that adds frequency-domain perturbations to protect speech against synthesis attacks, ensuring high privacy and speech quality.
Contribution
It proposes a novel universal perturbation approach with data augmentation and noise smoothing to enhance robustness and practicality in speech privacy preservation.
Findings
Effective against multiple speech synthesis models
Maintains high speech quality and utility
Supports universal protection for any speech length
Abstract
Recently, speech assistant and speech verification have been used in many fields, which brings much benefit and convenience for us. However, when we enjoy these speech applications, our speech may be collected by attackers for speech synthesis. For example, an attacker generates some inappropriate political opinions with the characteristic of the victim's voice by obtaining a piece of the victim's speech, which will greatly influence the victim's reputation. Specifically, with the appearance of some zero-shot voice conversion methods, the cost of speech synthesis attacks has been further reduced, which also brings greater challenges to user voice security and privacy. Some researchers have proposed the corresponding privacy-preserving methods. However, the existing approaches have some non-negligible drawbacks: low transferability and robustness, high computational overhead. These…
| Dataset | TTS Model | ECAPA | ResNet | ERes2Net | CAM++ | MOS | CTC-LOSS | CER | WER | ||||
| STCMR | STCS | STCMR | STCS | STCMR | STCS | STCMR | STCS | ||||||
| VoxCeleb1 | Glow-TTS | 0.97±0.00 | 0.08±0.01 | 0.99±0.00 | 0.09±0.01 | 1.00±0.00 | 0.11±0.01 | 1.00±0.00 | 0.14±0.01 | 3.21±0.23 | 3.65±1.37 | 0.00±0.00 | 0.00±0.00 |
| Speedy-Speech | 0.96±0.00 | 0.09±0.01 | 0.99±0.00 | 0.10±0.01 | 0.99±0.00 | 0.12±0.01 | 0.99±0.00 | 0.15±0.01 | 3.16±0.24 | ||||
| Tacotron2-DCA | 0.97±0.00 | 0.08±0.01 | 0.99±0.00 | 0.10±0.01 | 1.00±0.00 | 0.12±0.01 | 1.00±0.00 | 0.14±0.01 | 3.20±0.25 | ||||
| Tacotron2-DDC | 0.96±0.00 | 0.10±0.01 | 0.98±0.00 | 0.11±0.01 | 0.99±0.00 | 0.13±0.02 | 0.99±0.00 | 0.15±0.01 | 3.15±0.20 | ||||
| YourTTS | 0.97±0.00 | 0.09±0.01 | 0.99±0.00 | 0.10±0.01 | 1.00±0.00 | 0.13±0.02 | 1.00±0.00 | 0.14±0.02 | 3.19±0.22 | ||||
| LibriSpeech | Glow-TTS | 0.72±0.00 | 0.23±0.01 | 0.84±0.00 | 0.20±0.01 | 0.95±0.00 | 0.22±0.01 | 0.81±0.00 | 0.27±0.01 | 3.21±0.20 | 4.15±0.91 | 0.00±0.00 | 0.00±0.00 |
| Speedy-Speech | 0.73±0.00 | 0.23±0.01 | 0.85±0.00 | 0.20±0.01 | 0.95±0.00 | 0.22±0.01 | 0.82±0.00 | 0.26±0.01 | 3.19±0.19 | ||||
| Tacotron2-DCA | 0.67±0.00 | 0.24±0.01 | 0.81±0.00 | 0.22±0.01 | 0.91±0.00 | 0.24±0.01 | 0.74±0.00 | 0.29±0.01 | 3.17±0.20 | ||||
| Tacotron2-DDC | 0.71±0.00 | 0.24±0.01 | 0.84±0.00 | 0.21±0.01 | 0.91±0.00 | 0.24±0.01 | 0.79±0.00 | 0.28±0.01 | 3.17±0.19 | ||||
| YourTTS | 0.71±0.00 | 0.22±0.01 | 0.84±0.00 | 0.20±0.01 | 0.91±0.00 | 0.24±0.01 | 0.80±0.00 | 0.26±0.01 | 3.23±0.22 | ||||
| SVM | TTS Model | ECAPA | ResNet | ERes2Net | CAM++ | MOS | CTC-LOSS | CER | WER | ||||
| STCMR | STCS | STCMR | STCS | STCMR | STCS | STCMR | STCS | ||||||
| CAM++ | Glow-TTS | 0.80±0.00 | 0.23±0.01 | 0.87±0.00 | 0.24±0.01 | 0.92±0.00 | 0.28±0.01 | 0.90±0.00 | 0.31±0.01 | 3.20±0.24 | 3.78±1.67 | 0.00±0.00 | 0.00±0.00 |
| Speedy-Speech | 0.78±0.00 | 0.22±0.01 | 0.86±0.00 | 0.23±0.01 | 0.88±0.00 | 0.27±0.01 | 0.88±0.00 | 0.30±0.01 | 3.26±0.24 | ||||
| Tacotron2-DCA | 0.83±0.00 | 0.22±0.01 | 0.87±0.00 | 0.24±0.01 | 0.90±0.00 | 0.28±0.01 | 0.89±0.00 | 0.31±0.01 | 3.31±0.28 | ||||
| Tacotron2-DDC | 0.82±0.00 | 0.22±0.01 | 0.86±0.00 | 0.25±0.01 | 0.91±0.00 | 0.28±0.01 | 0.89±0.00 | 0.31±0.01 | 3.27±0.23 | ||||
| YourTTS | 0.84±0.00 | 0.20±0.01 | 0.89±0.00 | 0.22±0.01 | 0.92±0.00 | 0.28±0.01 | 0.92±0.00 | 0.30±0.01 | 3.33±0.33 | ||||
| ECAPA | Glow-TTS | 0.91±0.00 | 0.18±0.01 | 0.93±0.00 | 0.18±0.01 | 0.97±0.00 | 0.22±0.01 | 0.97±0.00 | 0.26±0.01 | 3.17±0.27 | 3.82±2.54 | 0.00±0.00 | 0.00±0.00 |
| Speedy-Speech | 0.90±0.00 | 0.18±0.01 | 0.94±0.00 | 0.18±0.01 | 0.97±0.00 | 0.22±0.01 | 0.96±0.00 | 0.25±0.01 | 3.15±0.21 | ||||
| Tacotron2-DCA | 0.90±0.00 | 0.19±0.01 | 0.93±0.00 | 0.19±0.01 | 0.96±0.00 | 0.23±0.01 | 0.98±0.00 | 0.26±0.01 | 3.24±0.25 | ||||
| Tacotron2-DDC | 0.89±0.00 | 0.19±0.01 | 0.92±0.00 | 0.19±0.01 | 0.97±0.00 | 0.22±0.01 | 0.95±0.00 | 0.25±0.01 | 3.16±0.19 | ||||
| YourTTS | 0.91±0.00 | 0.17±0.01 | 0.94±0.00 | 0.18±0.01 | 0.96±0.00 | 0.22±0.01 | 0.96±0.00 | 0.25±0.01 | 3.23±0.28 | ||||
| ERes2Net | Glow-TTS | 0.72±0.00 | 0.23±0.01 | 0.76±0.00 | 0.25±0.01 | 0.83±0.00 | 0.29±0.01 | 0.83±0.00 | 0.32±0.01 | 3.30±0.32 | 3.89±2.47 | 0.00±0.00 | 0.00±0.00 |
| Speedy-Speech | 0.72±0.00 | 0.23±0.01 | 0.77±0.00 | 0.24±0.01 | 0.85±0.00 | 0.29±0.01 | 0.84±0.00 | 0.31±0.01 | 3.23±0.26 | ||||
| Tacotron2-DCA | 0.72±0.00 | 0.22±0.01 | 0.78±0.00 | 0.25±0.01 | 0.86±0.00 | 0.29±0.01 | 0.85±0.00 | 0.31±0.01 | 3.31±0.32 | ||||
| Tacotron2-DDC | 0.75±0.00 | 0.22±0.01 | 0.79±0.00 | 0.25±0.01 | 0.87±0.00 | 0.29±0.01 | 0.85±0.00 | 0.31±0.01 | 3.28±0.27 | ||||
| YourTTS | 0.80±0.00 | 0.21±0.01 | 0.84±0.00 | 0.23±0.01 | 0.87±0.00 | 0.28±0.01 | 0.86±0.00 | 0.30±0.01 | 3.37±0.37 | ||||
| ResNet | Glow-TTS | 0.81±0.00 | 0.20±0.01 | 0.86±0.00 | 0.22±0.01 | 0.91±0.00 | 0.26±0.01 | 0.92±0.00 | 0.29±0.01 | 3.20±0.29 | 3.74±1.26 | 0.00±0.00 | 0.00±0.00 |
| Speedy-Speech | 0.82±0.00 | 0.20±0.01 | 0.88±0.00 | 0.22±0.01 | 0.92±0.00 | 0.26±0.01 | 0.91±0.00 | 0.28±0.01 | 3.27±0.32 | ||||
| Tacotron2-DCA | 0.83±0.00 | 0.20±0.01 | 0.86±0.00 | 0.23±0.01 | 0.92±0.00 | 0.27±0.01 | 0.94±0.00 | 0.29±0.01 | 3.32±0.32 | ||||
| Tacotron2-DDC | 0.81±0.00 | 0.21±0.01 | 0.87±0.00 | 0.23±0.01 | 0.92±0.00 | 0.27±0.01 | 0.93±0.00 | 0.30±0.01 | 3.32±0.30 | ||||
| YourTTS | 0.85±0.00 | 0.20±0.01 | 0.86±0.00 | 0.22±0.01 | 0.96±0.00 | 0.27±0.01 | 0.94±0.00 | 0.29±0.01 | 3.28±0.31 | ||||
| Data Augmentation | Noise Smoothing | ECAPA | ResNet | ERes2Net | CAM++ | MOS | CTC-LOSS | CER | WER | ||||
| STCMR | STCS | STCMR | STCS | STCMR | STCS | STCMR | STCS | ||||||
| FALSE | FALSE | 0.84±0.00 | 0.17±0.01 | 0.97±0.00 | 0.15±0.01 | 0.97±0.00 | 0.19±0.01 | 0.91±0.00 | 0.20±0.01 | 3.37±0.21 | 4.38±0.58 | 0.00±0.00 | 0.00±0.00 |
| TRUE | FALSE | 0.90±0.00 | 0.16±0.01 | 0.98±0.00 | 0.14±0.01 | 0.99±0.00 | 0.18±0.01 | 0.94±0.00 | 0.19±0.01 | 3.30±0.21 | 4.30±0.59 | 0.00±0.00 | 0.00±0.00 |
| FALSE | TRUE | 0.63±0.00 | 0.25±0.01 | 0.81±0.00 | 0.22±0.01 | 0.87±0.00 | 0.26±0.01 | 0.78±0.00 | 0.28±0.01 | 3.30±0.22 | 4.21±1.14 | 0.00±0.00 | 0.00±0.00 |
| TRUE | TRUE | 0.70±0.00 | 0.23±0.01 | 0.82±0.00 | 0.21±0.01 | 0.91±0.00 | 0.25±0.01 | 0.79±0.00 | 0.27±0.01 | 3.23±0.22 | 4.15±0.91 | 0.00±0.00 | 0.00±0.00 |
| SVM | Metric | Noise Frame Length=30 | Noise Frame Length=120 | ||||||||
| 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | 0.1 | 0.2 | 0.3 | 0.4 | 0.5 | ||
| ECAPA | Train-STCMR | 0.84±0.00 | 0.72±0.00 | 1.00±0.00 | 0.77±0.00 | 1.00±0.00 | 0.27±0.00 | 0.62±0.00 | 0.83±0.00 | 0.93±0.00 | 0.78±0.00 |
| Train-STCS | 0.20±0.01 | 0.25±0.01 | 0.13±0.00 | 0.22±0.01 | 0.16±0.01 | 0.34±0.01 | 0.26±0.01 | 0.21±0.01 | 0.16±0.01 | 0.21±0.00 | |
| Test-STCMR | 0.86±0.00 | 0.72±0.00 | 1.00±0.00 | 0.86±0.00 | 1.00±0.00 | 0.13±0.00 | 0.72±0.00 | 0.82±0.00 | 0.96±0.00 | 0.86±0.00 | |
| Test-STCS | 0.23±0.01 | 0.26±0.01 | 0.14±0.00 | 0.22±0.00 | 0.14±0.01 | 0.38±0.01 | 0.27±0.01 | 0.23±0.01 | 0.17±0.00 | 0.21±0.00 | |
| ResNet | Train-STCMR | 0.86±0.00 | 0.80±0.00 | 1.00±0.00 | 0.93±0.00 | 1.00±0.00 | 0.31±0.00 | 0.65±0.00 | 0.84±0.00 | 0.93±0.00 | 0.92±0.00 |
| Train-STCS | 0.21±0.01 | 0.23±0.01 | 0.13±0.00 | 0.23±0.00 | 0.16±0.01 | 0.35±0.01 | 0.28±0.01 | 0.24±0.01 | 0.17±0.01 | 0.21±0.01 | |
| Test-STCMR | 0.89±0.00 | 0.89±0.00 | 1.00±0.00 | 0.89±0.00 | 1.00±0.00 | 0.27±0.00 | 0.62±0.00 | 0.82±0.00 | 1.00±0.00 | 0.93±0.00 | |
| Test-STCS | 0.23±0.01 | 0.24±0.01 | 0.14±0.01 | 0.22±0.01 | 0.13±0.00 | 0.37±0.01 | 0.29±0.01 | 0.23±0.01 | 0.17±0.01 | 0.22±0.01 | |
| ERes2Net | Train-STCMR | 0.96±0.00 | 0.93±0.00 | 1.00±0.00 | 0.96±0.00 | 1.00±0.00 | 0.60±0.00 | 0.89±0.00 | 0.96±0.00 | 0.96±0.00 | 0.95±0.00 |
| Train-STCS | 0.22±0.01 | 0.25±0.00 | 0.12±0.01 | 0.23±0.01 | 0.18±0.01 | 0.36±0.01 | 0.29±0.01 | 0.26±0.01 | 0.19±0.01 | 0.25±0.01 | |
| Test-STCMR | 0.89±0.00 | 0.96±0.00 | 1.00±0.00 | 1.00±0.00 | 1.00±0.00 | 0.48±0.00 | 0.93±0.00 | 0.96±0.00 | 1.00±0.00 | 1.00±0.00 | |
| Test-STCS | 0.24±0.01 | 0.27±0.01 | 0.12±0.00 | 0.23±0.00 | 0.17±0.00 | 0.39±0.01 | 0.29±0.01 | 0.27±0.01 | 0.22±0.00 | 0.24±0.01 | |
| CAM++ | Train-STCMR | 0.95±0.00 | 0.87±0.00 | 1.00±0.00 | 0.93±0.00 | 1.00±0.00 | 0.62±0.00 | 0.77±0.00 | 0.90±0.00 | 0.95±0.00 | 0.92±0.00 |
| Train-STCS | 0.25±0.00 | 0.26±0.01 | 0.16±0.00 | 0.25±0.01 | 0.20±0.01 | 0.35±0.01 | 0.31±0.01 | 0.28±0.01 | 0.24±0.01 | 0.26±0.01 | |
| Test-STCMR | 0.93±0.00 | 0.96±0.00 | 1.00±0.00 | 0.96±0.00 | 1.00±0.00 | 0.34±0.00 | 0.86±0.00 | 0.93±0.00 | 0.96±0.00 | 0.86±0.00 | |
| Test-STCS | 0.27±0.00 | 0.27±0.01 | 0.18±0.00 | 0.26±0.00 | 0.19±0.00 | 0.40±0.01 | 0.31±0.01 | 0.28±0.01 | 0.25±0.01 | 0.27±0.01 | |
| MOS | 2.74±0.03 | 2.53±0.02 | 2.25±0.02 | 2.50±0.01 | 2.17±0.01 | 3.00±0.09 | 2.84±0.05 | 2.87±0.03 | 2.62±0.01 | 2.81±0.02 | |
| CER | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | |
| WER | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | 0.00±0.00 | |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSpeech Recognition and Synthesis · Voice and Speech Disorders · Adversarial Robustness in Machine Learning
FreeTalk:A plug-and-play and black-box defense against
speech synthesis attacks
Yuwen Pu, Zhou Feng, Chunyi Zhou, Jiahao Chen, Chunqiang Hu, Haibo Hu, Shouling Ji Yuwen Pu, Chunqiang Hu and Haibo Hu are with the School of Big Data & Software Engineering, Chongqing University, Chongqing, 400030, China. E-mail: {yw.pu, chu, haibo.hu}@cqu.edu.cn.Zhou Feng, Chunyi Zhou, Jiahao Chen and Shouling Ji are with the College of Computer Science and Technology at Zhejiang University, Hangzhou, Zhejiang, 310027, China. E-mail: {zhou.feng, zhouchunyi, xaddwell, sji}@zju.edu.cnYuwen Pu and Zhou Feng are the co-first authors.
Abstract
Recently, speech assistant and speech verification have been used in many fields (such as smart home, finance, speech translation, and audio navigation), which brings much benefit and convenience for us. However, when we enjoy these speech applications, our speech may be collected by attackers for speech synthesis. For example, an attacker generates some inappropriate political opinions with the characteristic of the victim’s voice by obtaining a piece of the victim’s speech, which will greatly influence the victim’s reputation. Specifically, with the appearance of some zero-shot voice conversion methods, the cost of speech synthesis attacks has been further reduced, which also brings greater challenges to user voice security and privacy. Some researchers have proposed the corresponding privacy-preserving methods. However, the existing approaches have some on-negligible drawbacks: low transferability and robustness, high computational overhead. These deficiencies seriously limit the existing method deployed in practical scenarios. Therefore, in this paper, we proposed a lightweight, robust, plug-and-play privacy preservation method against speech synthesis attacks in a black-box setting. Our method generates and adds a frequency-domain perturbation to the original speech to achieve privacy protection and high speech quality. Then, we present data augmentation strategy and noise smoothing mechanism to improve the robustness of the proposed method. Besides, to reduce the user’s defense overhead, we also proposed a novel identity-wise protection mechanism. It can generate a universal perturbation for one speaker and support privacy preservation for speech of any length. Finally, we conduct extensive experiments on 5 speech synthesis models, 5 speech verification models, 1 speech recognition model, and 2 datasets. The experimental results demonstrate that our method has satisfying privacy-preserving performance, high speech quality, and utility.
Index Terms:
Speech Protection, Privacy Preservation, Speech Recognition, Voice Conversion, Text-to-Speech.
I Introduction
Recent advances in speech synthesis have led to powerful techniques like Voice Conversion (VC) [1, 2] and Text-to-Speech (TTS) [3]. VC transforms the speaker’s voice characteristics while preserving linguistic content[4], whereas TTS generates natural speech from text input using a target speaker’s voice. These technologies are widely applied in voice assistants, language translation, and assistive communication tools [5, 6, 7, 8].
While speech synthesis technologies provide considerable convenience in human-computer interaction, they simultaneously raise critical concerns regarding security vulnerabilities and privacy infringements. A growing concern involves malicious actors exploiting these systems to fabricate artificial voices for fraudulent or infringing activities [9, 10]. Notably, emerging zero-shot VC techniques [11, 12, 13] enable attackers to synthesize highly realistic voice replicas using merely seconds of victim audio samples obtained from social media platforms. Such technological advancements have led to increasingly sophisticated attack vectors, as exemplified by a case where synthesized CEO voice audio defrauded victims of over \243,000$ through telephone deception [14, 15].
To counter these emerging threats, current research has primarily focused on developing adversarial-based defense mechanisms [15, 3, 9, 10]. They mainly protect user’s voice from speech synthesis attacks by adding some adversarial perturbations to original audio. Although these defensive methods have achieved some success in combating voice spoofing, they often suffer from these significant drawbacks. (i) Low Transferability and Strong Assumptions. Current defense methods assume knowledge of the attack model, an unrealistic assumption given that users cannot know which specific model surveillants use in practice [9, 3, 15]. Consequently, these adversarial perturbations demonstrate limited cross-model transferability, although they may perform better on known attack models - an overly strong assumption in practical scenarios. (ii) Limited Generalizability. Existing methods require unique computations for each speech segment [3, 15], fundamentally restricting their generalizability across different usage scenarios. This segment-specific approach results in poor scalability across input durations, as the framework cannot effectively generalize its protective mechanisms to varying waveform lengths. The consequent computational inefficiency emerges as a direct manifestation of this core limitation, particularly evident in resource-constrained deployment environments where generalizable solutions are paramount.
Motivated by these gap, we introduce FreeTalk, which is a plug-and-play and black-box defense approach against speech synthesis attacks. Using FreeTalk, although the processed sample maintains perceptual similarity to the victim’s voice for human listeners, synthesis using this sample by attacker generates speech that diverges from the victim’s vocal characteristics. Consequently, the synthesized speech cannot deceive authentication services or the victim’s acquaintances, while preserving the victim’s ability to share speech normally on voice assistant platforms. FreeTalk can be seamlessly deployed on major voice assistant platforms as a plug-and-play solution, providing universal audio protection without compromising perceptual quality. This enables users to upload audio content freely and securely without concerns over privacy or auditory degradation.
Designing FreeTalk presents three main technical challenges:
Transferability. Considering the limited background knowledge of practical defenders, it is difficult for them to determine the type of the attack model, making it challenging for perturbation methods to achieve cross-model transferability. How can we generate perturbations with good transferability against unknown attack types in black-box scenarios? 2. 2.
Robustness. Current methods, while ensuring security by adding noise to audio, often significantly degrade usability (e.g., timbre). However, overly small perturbations provide insufficient protection. How can we strike an optimal trade-off, ensuring robust speech protection while maintaining high-precision speech recognition? 3. 3.
Generalizability. Current methods predominantly apply noise injection to individual speech segments, incurring substantial computational overhead - particularly when processing lengthy protected speech. How can we design an efficient identity-wise perturbation mechanism that produces universal noise patterns applicable to all speech instances from a given user?
To overcome these challenges, we introduce the first plug-and-play and black-box defense against speech synthesis, namely FreeTalk. For challenge 1), FreeTalk equips with a novel sample-wise perturbation mechanism that applies frequency-domain perturbations to original speech signals, achieving improving transferability in the context of black-box and unknowing attack scenarios. For challenge 2), FreeTalk combines a novel data augmentation strategy with an advanced noise smoothing mechanism, specifically designed to eliminate high-frequency distortions while preserving the spectral integrity of protected speech signals. For challenge 3), FreeTalk introduces a hierarchical patch optimization method combining identity-wise and frame-wise processing, enabling both speaker-universal perturbation generation and length-agnostic speech preservation. The specific contributions of this paper are outlined as below:
We propose FreeTalk, the first black-box, plug-and-play speech protection framework that effectively defends against speech synthesis attacks. The framework could demonstrate strong transferability against unknown attack models. 2. 2.
We design a novel data augmentation strategy to enhance robustness with a specialized noise smoothing mechanism. It effectively eliminates high-frequency distortions while preserving the spectral integrity of protected speech signals. 3. 3.
We present a hierarchical patch optimization method that integrates identity-wise and frame-wise processing to enhance generalizability, enabling both speaker-universal perturbation generation and length-agnostic speech preservation. 4. 4.
We evaluate our FreeTalk on 5 speech synthesis models, 5 speech verification models, 1 speech recognition model, and 2 datasets. Experimental results demonstrate that FreeTalk achieves strong privacy protection while maintaining high speech quality and utility.
II Related Works
In this section, we review the relevant works on speech synthesis methods and introduce the development of voice conversion defense methods.
II-A Speech Synthesis Attacks
In speech synthesis attacks, the attacker aims to generate a fake speech with the target speaker’s voice identity, speaking some specific words selected by the attacker. In this paper, we mainly focus on two popular speech synthesis attacks, including VC and TTS.
II-A1 Voice Conversion
VC can be categorized into parallel VC and nonparallel VC based on whether they require a parallel corpus for training. In the early stages of VC development, many parallel VC approaches are proposed [16, 17, 18]. For example, based on the transformer architecture, Liu et al. [19] proposed a VC model that can generate high quality speech with high similarity to the target speaker. Helander et al. [20] proposed a VC approach by combining the partial least squares (PLS) with the Gaussian mixture model (GMM), which allows the use of multiple local linear mappings. However, in the real scenario, it is impractical to obtain a number of parallel training data. Therefore, many non-parallel VC methods are proposed[21, 22, 23, 24, 25]. For example, Liu et al. [1] designed a robust nonparallel and supporting any-to-any VC method. This method can achieve satisfactory performance in terms of naturalness and target speaker similarity. Kang et al. [13] presented an end-to-end zero-shot voice conversion model that can disentangle content and timbre information and then efficiently synthesize the time domain audio. Nguyen et al. [26] devised an end-to-end adversarial network that supported conducting VC directly on arbitrary length raw audio waveform and had a fine performance. Wang et al. [27] proposed a large-model-based zero-shot VC method that used a two-stage framework. FreeTalk can easily generate the fake speech with only 3 seconds of the target speaker’s voice.
II-A2 Text-to-Speech
Similar to VC, TTS generates a target utterance with arbitrary texts and the timbre of the target speaker. Now, many advanced TTS methods are also proposed [28, 29, 30]. For example, Wang et al. [31] introduced a language model approach for TTS with audio codec codes as intermediate representations. This approach can synthesize high-quality personality speech with only a few seconds speech from the target speaker. Shen et al. [11] designed an efficient TTS system that uses a neural audio codec with continuous latent vectors to achieve natural and zero-shot TTS synthesis. Huang et al. [12] proposed a TTS model that disentangled the speech into style-agnostic and style-specific information. This model had a good performance with high audio quality and style similarity.
II-B Defending against Speech Synthesis Attacks
To mitigate speech synthesis attacks, many defense methods have been proposed, including two directions: post-detection [32, 33, 34, 35] and preventing the attacker from generating synthetic speeches [15, 36, 37, 38]. Note that in this paper, we mainly focus on how to protect speakers’ voice identity privacy by preventing the attacker to generate synthetic speeches. There are some relevant privacy-preserving methods that mainly employ the idea of adversarial example generation. For example, Huang et al. [37] prevented the speaker’s voice characteristics from leaking by introducing some human imperceptible noise into the utterances of a speaker. Chen et al. [9] proposed a speaker protection method that perturbs speech signals by minimally altering the original speech while preventing the attacker from accurately generating the voice of the target speaker. Note that both of the two above methods perform well only in the white-box scenario. Some methods are also proposed for both white-box and black-box scenarios. For example, Liu et al. [15] proposed two defense methods that can significantly degrade the performance of speech synthesis attacks while maintaining the sound of the speaker’s voice, but the two defense methods have to obtain the output of the attack model given an input. Yu et al. [3] designed a defense mechanism that relies on the transferability of adversarial examples to prevent unauthorized speech synthesis, but assumed that the attack model must share similarities with other robust encoders for improved efficacy. Li et al. [39] proposed a voice guard defense method that added the adversarial perturbation to the time domain. Dong et al. [40] presented a Generative Adversarial Network (GAN) framework to defend against malicious speech synthesis, which had a fine defense effectiveness and generation time. However, all the above methods perform well in the white-box setting and have poor performance in black-box. [36] and [41] also proposed to perturb the speaker’s voice to make them unlearnable for other deep-learning-based audio models. Note that all existing methods must generate a specific perturbation for each speech of the same speaker, which significantly increases computational overhead.
III Threat Model
III-A Attacker Motivation and Assumptions
Attack Motivation: The attacker intends to leverage the victim’s voice samples, which are uploaded to voice assistant tools (such as speech-to-text converters, translators, etc.), to synthesize speech using TTS tools. By imitating the victim’s vocal characteristics, the attacker can generate speech with manipulated content, thereby engaging in malicious activities such as fraudulent blackmail or reputational damage, ultimately causing severe harm to the victim.
(1) Financial Motivation. Attackers primarily seek monetary gain through voice spoofing, including financial fraud (e.g., fraudulent transfers), ransom extortion, and bypassing voice authentication systems in banking or corporate environments. Commercial espionage may also occur through executive voice impersonation, as exemplified by a case where synthesized CEO voice audio defrauded victims of over \243,000$ through telephone deception [14, 15].
(2) Sociopolitical Motivation. This involves manipulating public perception by fabricating damaging voice content to harm reputations of individuals or organizations. Politically, such attacks can spread disinformation by impersonating public figures to influence elections or international relations. During the Russia-Ukraine conflict, a fabricated audio of President Zelensky “ordering soldiers to surrender” circulated on social media to undermine Ukrainian troops’ morale [42].
(3) Personal Motivation. Attackers pursue personal objectives like revenge or harassment through targeted voice manipulation. This includes creating threatening messages or fabricating emotional distress scenarios to exploit personal relationships.
Attacker’s Assumptions: We assume that the attacker can obtain some victim users’ voice snippets, such as by eavesdropping from voice assistant tools, which is relatively easy to achieve because numerous existing studies have demonstrated privacy leakage vulnerabilities in current voice-based systems[43, 44]. Alternatively, the attacker may directly collect voice samples if they have a prior acquaintance with the victim. The attacker also has enough computational resource to run the SOTA speech synthesis methods (e.g., zero-shot speech synthesis) to generate fake speech of the target speaker.
III-B System Goals and Defender Assumptions
Defender’s Goal: The defender employs voice assistant tools or voice translator for services. To defend against potential threats, FreeTalk applies security processing to the uploaded speech. This ensures that automatic speech recognition and other voice-based services function normally while preventing attackers from synthesizing high-value voice samples that mimic the victim’s timbre using voice generation tools.
Defender’s Assumptions: The defender possesses no prior knowledge of the attacker’s methodologies or models. The defender’s capability is limited to processing the shared speech using the proposed FreeTalk prior to public release.
IV Design of FreeTalk
In this section, we propose a sample-wise privacy-preserving method that can generate a specific perturbation for a piece of speech. Then, to ensure the robustness of the perturbation and speech quality, we also propose data augmentation strategy and noise smoothing mechanism to enhance the presented method. Furthermore, we present a novel identity-wise privacy protection method that can reduce computational overhead and support any length from the same speaker.
IV-A Sample-wise Optimization for Protection
Our proposed algorithm addresses unauthorized voice cloning by introducing frequency domain perturbations that preserve speech intelligibility while disrupting the extraction of speaker-specific features. This is motivated by the observation that TTS systems primarily rely on specific frequency bands to extract speaker characteristics [45, 46], making these bands ideal targets for protective perturbation. Since speaker characteristics are predominantly encoded in specific frequency bands of the speech signal [47], here we strategically introduce controlled perturbations in these frequency ranges. In this way, we can effectively protect the speaker’s identity while maintaining the natural quality of the speech. The core protection strategy involves the following key components.
Given an input waveform , we first extract its speaker embedding using a pre-trained speaker verification model. This embedding serves as a reference for measuring the effectiveness of our protection mechanism. Next, we compute its Short-Time Fourier Transform (STFT) representation for the frequency feature, where and denote frequency bins and time frames, respectively. The detailed parameters of the STFT used in this paper are presented in V.A.6). The frequency band ( and ) for perturbation is determined by a ratio parameter , which helps to identify the most relevant frequency components for the characteristics of the speaker. Within this band, we establish a masking threshold based on the minimum magnitude of the STFT coefficients:
[TABLE]
which ensures that perturbations are applied only to significant spectral components.
Further, we initialize the complex noise where represent the real and imaginary components, respectively, denoting the perturbation on the frequency domain. The noise is refined through iterations using gradient descent. At each iteration, we apply a frequency mask defined as:
[TABLE]
This mask ensures that perturbations are concentrated in the target frequency band and applied only to significant spectral components. The protected waveform is generated by applying the masked noise to the original spectrogram: , where controls the perturbation magnitude. The effectiveness is evaluated by comparing the original embedding with the embedding extracted from the augmented protected waveform (discussed in Sec.IV-B). The noise is updated using Adam optimizer to maximize the embedding distance while maintaining signal integrity using noise smoothing (discussed in Sec.IV-C) before applying the frequency perturbations. Our approach incorporates data augmentation and noise smoothing techniques to enhance robustness against various audio transformations. The augmentation (which might degrade audio quality) ensures protection persists under common audio processing operations while smoothing (which might sacrifice effectiveness) prevents artifacts and maintains natural sound quality. The iterative optimization process allows for fine-tuned control over the protection-intelligibility trade-off through parameters and , while the frequency-selective approach ensures minimal impact on the original speech content (the complete algorithm is presented in Algorithm. 1). This makes our method particularly suitable for real-world applications where both privacy protection and speech quality are crucial.
IV-B Augmentation
The data augmentation strategy for waveform signals consists of three sequential transformations to enhance the robustness of protection. Gaussian Noise Injection: First, we add random gaussian noise to simulate environmental interference and prevent overfitting: . This helps the perturbation become more resilient to background noise in real-world scenarios. Time Shifting: We then apply a random temporal shift to make the perturbation invariant to the exact timing of speech segments:
[TABLE]
where . This transformation ensures that the model focuses on the speech content rather than its absolute position in time. Volume Scaling: Finally, we apply random amplitude scaling to account for varying speech volumes: , where . This makes the model more robust to variations in speaking volume and recording conditions. The final augmented waveform incorporates all these transformations, creating a more challenging and diverse training signal. This comprehensive augmentation strategy helps prevent overfitting and improves the model’s generalization ability across different acoustic conditions, speaking styles, and recording environments. The carefully chosen parameters (e.g., noise variance of 0.01, shift range of ±10%, and scaling range of 0.8-1.2) provide meaningful perturbations while preserving the essential characteristics of the speech signal.
IV-C Noise Smoothing
The noise smoothing mechanism is designed to reduce high-frequency artifacts and ensure spectral consistency in the frequency domain. Given complex noise , we apply a one-dimensional averaging filter: (1) Kernel Definition. We first define a normalized averaging kernel: , where we use in this paper. This kernel performs local averaging over a window of 5 time steps. (2) Smoothing Operation. The smoothing is applied separately to real and imaginary components through valid convolution:
[TABLE]
where denotes the convolution operation with appropriate padding to maintain the original dimensions. Thus we have . This smoothing process serves multiple purposes: (1) Ensure temporal continuity in the perturbation signal; (2) Reduce potential artifacts that could arise from abrupt noise changes; (3) Maintain the spectral structure of the original signal while applying adversarial perturbations; (4) Help create more natural-sounding modifications by avoiding sharp transitions.
The smoothed noise is then applied to the original STFT representation with proper masking and scaling, resulting in more coherent and perceptually acceptable modifications to the audio signal. The kernel size of 5 provides a good balance between smoothing effectiveness and preservation of local temporal structure.
IV-D Identity-wise Optimization for Protection
While sample-wise protection provides effective defense against voice cloning, it requires generating unique perturbations for each audio sample, which may be computationally intensive in real-world applications. To address this limitation, we propose an identity-wise protection mechanism that learns a universal perturbation pattern for each speaker identity, significantly reducing computational overhead while maintaining robust protection. Moreover, considering that the length of different waveforms varies as well, we introduce an identity-wise and frame-wise patch optimization strategy, enabling the preservation of waveforms with any length.
Our approach is motivated by the observation that speaker characteristics remain relatively consistent across different utterances from the same speaker. This consistency allows us to develop a single, optimized frequency perturbation that can effectively protect all audio samples from a given speaker. Given a training set containing multiple utterances from the same speaker, we learn a universal complex noise pattern that generalizes across all samples. The algorithm consists of two main components.
Universal Noise Learning. Similarly, we initialize identity-wise complex noise components and optimize them iteratively. For each training iteration, instead of applying direct gradient descent, we employ a PGN [48]-inspired two-stage interpolation strategy to better what. Specifically, for each audio sample , we first sample a perturbed noise pair from a local neighborhood and compute the first-stage gradient . Then, we perform a forward projection to generate , and obtain the second-stage gradient with the updated protection. The final gradient is interpolated as:
[TABLE]
which is accumulated across all training samples. This interpolation strategy stabilizes the optimization and encourages the learned universal noise to be more transferable and robust.
Frame-wise Noise Application. To enhance robustness and reduce artifacts, we introduce a novel frame-wise noise application strategy. Specifically, we reformulate the perturbation into the tiled frame patch, which is applicable for waveforms of any length. Given frame length (patch length) and mask ratio , we first divide the STFT spectrogram into frames. Then, a random binary mask is generated where each frame has probability of being selected:
[TABLE]
With the mask, we apply the universal noise selectively to chosen frames: , where and define frame boundaries.
Several key parameters control the effectiveness of our method: the noise level affecting protection strength, the mask ratio controlling the density of protection, and the frame length of patch determining the granularity of noise application. These parameters can be tuned to achieve an optimal balance between protection effectiveness and speech quality for different application scenarios, which will be further discussed in our ablation study. And to further smooth high-frequency artifacts, we apply a lightweight Wiener filter to the perturbed STFT, enhancing perceptual quality without degrading protection.
This frame-wise application strategy serves multiple purposes: (1) introduces temporal variability in protection, making it harder for attackers to reverse-engineer the protection; (2) reduces potential artifacts by maintaining clean frames; (3) allows for fine-grained control over the protection-quality trade-off through the mask ratio .
The overall process for identity-wise optimization is shown in Algorithm.2. This approach offers several advantages over sample-wise protection:
- •
Reduced computational cost during deployment, as the noise pattern needs to be computed only once per speaker.
- •
Consistent protection across all utterances with various lengths from the same speaker.
V Experiments and Evaluation
V-A Experimental Settings
All experiments were conducted on a server equipped with Intel(R) Xeon(R) Platinum 8358P CPUs, 3.40GHz processor, 386GB RAM, and NVIDIA A800. VSCode and PyTorch are used to deploy the model and complete relevant experiments.
V-A1 Speech Corpus Datasets
Because of the ethical considerations, we use public datasets (LibriSpeech and VoxCeleb1) to evaluate the effectiveness of FreeTalk.
LibriSpeech[49] is a corpus of read English speech which contains hours of speech sampled at kHz. It is suitable for training and evaluating speech recognition systems. LibriSpeech is frequently used to train and evaluate automatic speech recognition (ASR) models. Due to its large scale and accurate text annotations, it has become one of the standard datasets in ASR research.
VoxCeleb1[50] is an audio-visual dataset extracted from interview videos uploaded to YouTube, which contains more than hundreds of thousands of ’real world’ utterances for over celebrities. Due to its diversity of speakers and background noise, VoxCeleb1 is particularly valuable for developing robust speaker recognition systems. It is often used to benchmark the performance of speaker identification and verification models.
V-A2 Speech Synthesis Models
In our threat model, we assume that the defender has no knowledge about the speech synthesis models used by the attacker. Hence, to verify the defense performance of FreeTalk, we also employ five state-of-the-art speech synthesis models, including Tacotron2-DDC[51], Tacotron2-DCA[52], Glow-TTS[46], YourTTS[45], and Speedy-Speech[53].
Tacotron2-DDC[51] is proposed based on TacoTron2[54] that synthesizes time-domain waveforms by using a modified WaveNet vocoder. Tacotron2-DDC applys Dynamic Decode Convolution enhancement into decoder, which enables the model to capture both local and global contextual information more effectively, leading to improved speech synthesis quality.
Tacotron2-DCA[52] is a popular zero-shot speech synthesis model, which achieves better text alignment and smoother, natural speech synthesis by using an energy-based attention mechanism. This model can synthesize high-quality speech for victim speakers only with a few seconds of target audio.
Glow-TTS[46] is stream-based generation model specifically designed for parallel Text-To-Speech (TTS) conversion. It supports end-to-end alignment and faster inference to produce stable and consistent output.
YourTTS[45] is a zero-shot multi-speaker TTS and voice conversion model, which is suitable for high-quality cross-language speech synthesis. It can generate the synthetic audio with less than one minute voice of the victim.
Speedy-Speech[53] is a lightweight TTS model with fast inference speed, which is suitable for the real-time applications. It can be efficiently trained on a single GPU and can run in real time even on a CPU.
V-A3 Speech Verification Models
In our paper, we suppose that both speech verification (SV) model and speech synthesis model have a similar function (feature extraction) as an encoder. Therefore, we use a speech verification model (e.g., X-Vector[55, 56]) to optimize the protected victim’s audio. Moreover, to verify whether a synthesis voice based on the protected audio matches the victim’s original audio, we also employ four pre-trained speech verification models (ECAPA[55, 56], ResNet[55, 56], ERes2Net[57, 58], CAM++[57, 58]) to evaluate.
X-Vector[55, 56] is a classic DNN-based speaker verification model by using convolutional layers and average pooling for fixed-length embedding. Data augmentation (e.g., adding noises and reverberation) is also proposed to improve the model performance and robustness.
ECAPA[55, 56] is a Time Delay Neural Network (TDNN)-based speaker feature extractor, which puts more emphasis on channel attention, propagation and aggregation for high-precision feature extraction.
ResNet[55, 56] can capture rich speaker features, particularly robust in complex audio environments because of the residual network structure.
ERes2Net[57, 58] is the extended Res2Net with multi-scale feature learning, capturing speaker info effectively even in short segments.
CAM++[57, 58] is a speaker recognition model based on densely connected delay neural network, which uses CAM-Pooling Plus for adaptive feature aggregation, enhancing verification performance.
V-A4 Speech Recognition Model
To evaluate the impact of FreeTalk on speech recognition task, we use Whisper [59] to convert the original speech and protected speech to text, respectively. Whisper is a multilingual speech recognition model developed by OpenAI. Whisper is an end-to-end model with multilingual and multitasking capabilities, including speech-to-text, speech translation, and speaker recognition.
V-A5 Evaluation Metrics
In order to evaluate the privacy-preserving performance of our method, we use Source and TTS-Camouflage Waveforms Mismatch Rate (STCMR) and Source and TTS-Camouflage Waveforms Similarity (STCS) to show the difference between the source audio and the synthetic voice. Moreover, to evaluate the speech quality of the protected audio, we also employ Mean Option Score (MOS). Besides, to evaluate the utility of the protected audio, we also use Connectionist Temporal Classification Loss (CTC-LOSS), Character Error Rate (CER) and Word Error Rate (WER). The introduction of the evaluation metrics is shown as following.
STCMR is defined as the rate that the generated synthetic audio based on the protected voice can not be matched with the source audio of the original speaker after inputting the SV model, which likes the false recognition rate of the SV model. The value of STCMR is [math] to . A higher STCMR value represents a better privacy-preserving effectiveness, which indicates that the more difference between the source audio and the synthetic audio.
STCS is defined as the similarity between the synthetic audio and the source audio after feature extraction, whose scale is from [math] to . A low STCS denotes that the more difference between the source audio and the synthetic audio, which also shows a better defense performance.
MOS represents a ranking of the quality and naturalness of speech on a scale from [math] to . This metric is used to assess the overall audio quality of the protected audio. The higher MOS indicates the better quality of the generated audio.
CTC-LOSS is the Connectionist Temporal Classification Loss. We take the original audio as the input and the Whisper output as the real label. CTC-LOSS represents the difference between the protected audio after inputting the Whisper and the real label in the feature space. When the CTC-LOSS is less than 10, it indicates that the prediction result is similar to the real label. That is, the lower CTC-LOSS represents better quality speech.
CER is the Character Error Rate that represents the character difference between the original audio and the protected audio after being converted to text. The lower CER represents better quality speech.
WER is the Word Error Rate that represents the word difference between the original audio and the protected audio after being converted to text. The lower WER represents better quality speech.
Therefore, to ensure defense performance, speech quality, and audio utility simultaneously, our goal is to achieve higher STCMR, MOS, and lower STCS, CTC-LOSS, CER, WER.
V-A6 Parameters Configuration
In our experiments, we configure our method to optimize privacy-preserving speaker embeddings through a series of adversarial steps and data augmentations. The primary parameters for our experimental setup are as follows:
Details of STFT: We adopt Short-Time Fourier Transform (STFT) for frequency-domain processing of waveform audio. In all experiments, the following parameter settings are used:
- •
Sampling rate: 16,000 Hz
- •
FFT window size: 1024
- •
Hop length: 512
- •
Window length: 1024
- •
Window function: Hann window111The Hann window is a commonly used tapering function that reduces spectral leakage by smoothly weighting the ends of each frame towards zero.
The STFT is computed on single-channel audio. The same configuration is used for both STFT and inverse STFT operations. These settings ensure consistent time-frequency resolution and phase preservation throughout all frequency-domain manipulations.
Loss Function: We used cosine similarity as loss function, an approach that helps ensure that the resulting embedding differs from the original without needing a specific target. This untargeted setup supports general privacy protection without narrowing the embedding to match a particular output.
Optimization Steps: The model undergoes optimization steps to iteratively refine the waveform embeddings. Each step applies small perturbations to the waveform in the frequency domain.
Learning Rate: The learning rate is set to for the Adam optimizer to ensure controlled yet effective convergence during the refinement process.
Frequency Interval: We set the frequency interval to , enabling perturbations focused around the middle frequency range. This choice limits the noise impact on the signal while maximizing privacy in critical frequency bands.
Data Augmentation: Data augmentation is applied to enhance the robustness of the refined waveform. This augmentation includes minor noise additions, time shifting, and random scaling to simulate varied acoustic environments.
Noise Smoothing: A smoothing operation is applied to the perturbation noise, with a kernel size of , to reduce abrupt changes in the frequency domain. This noise smoothing enhances the perceptual quality.
Noise Level: We set the noise level to for each perturbation applied to the frequency spectrum. This controlled noise level balances embedding robustness and signal quality.
V-B *Overall Performance for *FreeTalk
In our study, we employ the X-Vector to generate the perturbation. Four speech synthesis models and four speech verification models are used to evaluate the privacy-preserving performance of the perturbation generated by our method. The overall performance of the proposed approach is shown in Table I. From the experimental results in Table I, we mainly have three observations.
(1) Privacy-preserving performance: we observe that the lowest STCMR exceeds and the highest STCS is not more than for the four speech synthesis models and four speech verification models on VoxCeleb1. It is difficult for speech verification models to recognize the synthetic audio and the original audio as the voice of the same person. This indicates that the synthetic audio generated based on the protected speech differs greatly from the original audio in feature space, demonstrating the effectiveness of our method. Moreover, we find that the protective performance of our method is somewhat different in different datasets. It is obvious that the protection performance is better on VoxCeleb1 rather than LibriSpeech. For example, the lowest STCMR values are and , respectively. We suspect the reason is that VoxCeleb1 contains more natural speech data from conversations, interviews, speeches, and other scenarios, making the speech content closer to everyday life and actual communication contexts, possibly including various background noises, filler words, pauses, etc., with richer and more diverse emotional expressions and language styles. This makes the optimized perturbations with better transferability. In contrast, the speech content of LibriSpeech mainly consists of readings of written language, with relatively standard and formal language expression, and a relatively single emotional tone, resulting in poorer transferability. Furthermore, the performance of FreeTalk can be seen in different voice authentication models. In addition, we observe that the protection performance has subtle differences in various speech verification models. For example, when facing YourTTS on the LibriSpeech, the STCMR is 0.71 and 0.91 on the ECAPA and ERes2Net,respectively. We guess that it is due to ERes2Net placing more emphasis on the fusion extraction of local and global features, integrating features within a single residual block to extract local signals, and aggregating different scale acoustic features from various hierarchical outputs as global signals, thereby obtaining more representative and distinctive speaker characteristics. In contrast, ECAPA utilizes a delay neural network framework to extract features at different time steps to capture the temporal information of speech, and then further filters and enhances key features through mechanisms such as SESqueeze Excitation Layer, making the speaker’s feature information less susceptible to disturbance. Finally, we can find that FreeTalk has a stable defense performance on different speech synthesis models. For example, the STCMR’s fluctuation range under different speech synthesis models is no more than when operating on VoxCeleb1 and CAM++.
(2) Speech quality: For audio protected by our method, we can see that the MOS is from to . It objectively shows the quality of the audio processed by our method.
(3) Speech utility: To demonstrate the utility of protected speech, we input the protected speech and the original speech into Whisper, respectively. From Table I, we can know that CTC-LOSS remains at a high value, indicating that the protected speech and the original speech are similar in the feature space after Whisper processing. Additionally, when converting the original speech and the protected speech into text using Whisper, it was found that both the character error rate and the word error rate are 0. This shows that protected speech has almost no impact on the speech conversion task and does not significantly compromise speech usability.
V-C Sensitivity Analysis
In this subsection, we evaluate various of factors that may influence the privacy-preserving performance of our method.
V-C1 Different noise levels
Our method has to generate the the the perturbation added to the frequency spectrum. The noise level of the perturbation may affect the performance of our method. Therefore, we generate perturbation based on the X-Vector and evaluate the defense performance with different noise levels when facing YourTTS on LibriSpeech. As shown in Fig. 3, with an increase in noise level, the value of STCMR and STCS maintains a general growth and decreases, respectively. It indicates that protection performance keeps improving as the noise level increases. However, the MOS is gradual decline, which indicates that the speech quality is deterioration. We infer that as the noise level increases, the noise has a greater effect on the audio. While the privacy-preserving is improved, the sound quality is also compromised. Moreover, for the speech utility, we can see that both CER and WER always maintain 0. It indicates that the different noise levels nearly have no influence on the speech utility.
V-C2 Different frequency intervals
Considering that our method needs to add a perturbation to the frequency domain of the source waveform, the value of the frequency interval may influence the performance of our method. Therefore, keeping the dataset, the speech synthesis model and the speech verification model consistent with those in the previous paragraph (different noise level), we evaluate the performance of our methods in different frequency intervals when the noise level is fixed at . The experimental results are shown in Fig. 4. From Fig. 4, we can see that with an increase in the frequency interval value, the STCMR and STCS continue to rise and fall, respectively. It indicates that the privacy-preserving effect is improving steadily. Moreover, the MOS drops gradually, which demonstrates that the audio quality gradually deteriorates. We guess that with the increasing of frequency interval, the window for adding noise becomes larger. For the YourTTS model, the features become more blurred, and this also may lead to a decrease in sound quality. Besides, CTC-LOSS is almost stable. Both CER and WER always maintain 0. It indicates that the different frequency intervals have almost no influence on the speech utility.
V-C3 Different speech verification models
In our method, the defender needs to use a speech verification model to assist in generating the perturbation. To verify the generalization of our method, we evaluated the privacy-preserving performance on four different models and VoxCeleb1 to optimize the perturbation. Other parameter settings are set the same as the overall experiments. The experimental results are shown in Table II. From Table II, we can know when the perturbation is generated based on different models, the STCMR and STCS have some fluctuations. For example, when facing Glow-TTS, the protected audio generated by ECAPA (STCMR:0.91±0.00, STCS:0.18±0.01) can achieve a more privacy-preserving performance than that generated by ERes2Net (STCMR:0.72±0.00, STCS:0.23±0.01). However, the lowest STCMR is still more than 0.72 and the highest STCS is less than 0.32. This indicates that our method has a good generalization on different models. In addition, for speech quality, we can see that the MOS is almost stable around 3.2. This demonstrates that the speech quality is almost unaffected by speech verification model. For speech utility, the CTC-LOSS is always around 3.8. Moreover, both CER and WER maintain 0. This indicates that the speech utility is not affected by different speech verification models in our method.
V-D Ablation Study
In our method, we employ data augmentation technology to enhance the model robustness and use the noise smoothing mechanism to improve the speech quality. Therefore, we analyze the impact of data augmentation and noise smoothing on our method. We generate a perturbation by using the X-Vector on the LibriSpeech. YourTTS is used to evaluate the effectiveness of our method. As shown in Table III, we can see that when only employing data augmentation technology, the STCMR increases and the STCS decreases. Meanwhile, the MOS also slightly declines. It indicates that using data augmentation can improve the privacy-preserving performance while reducing the audio quality in our method. Moreover, when only using the noise smoothing mechanism, the STCMR decreases, while the STCS increases. This demonstrates that the privacy-preserving effect is reduced while the audio quality improving when the noise smoothing mechanism is used. Hence, to balance the privacy-preserving performance and audio quality, both data augmentation technology and the noise smoothing mechanism are used in our method.
V-E Transferability Analysis
The existing methods have to generate a specific perturbation for each audio sample. In this paper, to reduce the computational overhead of the user, we proposed an identity-wise optimization protection method. In this section, we evaluate the performance of the proposed identity-wise optimization protection method. Moreover, we also consider two situations: the frame length is 30 and 120, respectively. Concretely, we use ECAPA to assist in generating the protected audio and employ YourTTS to evaluate the defense performance on LibriSpeech. We divided the corpus of one person into a training set and a test set. Then, we generate a universal perturbation patch based on the training dataset. Finally, we add the patch to the train set and the test set, respectively. Relevant indicators are also used to evaluate the effectiveness of FreeTalk. This section considers the noise size ranges from 0.1 to 0.5 and the noise frame length is 30 and 120, respectively. The relevant experimental results are shown in Table IV. Three conclusions can be drawn as follows:
(1) Privacy-preserving: Note that Train-STCMR and Test-STCMR represent the STCMR on the training set and test set, respectively. Train-STCS and Test-STCS represent the STCS on the training set and test set, respectively. From Table IV, when the noise frame length is 30, we can know that both Train-STCMR and Test-STCMR maintain around . This indicates that the synthetic audio and the original audio are identified as different people. Moreover, Train-STCS and Test-STCS are always stable from to , indicating that the synthetic audio and the original audio have great differences in feature space. Compared with the experimental results with noise frame length=30, when the noise frame length is 120, we can see that both Train-STCMR and Test-STCMR slightly decrease, Train-STCS and Test-STCS slightly increase. It indicates that privacy-preserving performance is better when noise frame length is 30. In addition, note that when the noise frame length is 120 and the noise level is 0.1 and 0.2, both Train-STCMR and Test-STCMR are significantly lower than the other results. We guess that when the noise frame length is 120, many samples were truncated. When the noise level is 0.1 and 0.2, respectively, the noise amplitude is relatively small. This has a significant impact on the effect of privacy protection. Overall, most of Train-STCMR and Test-STCMR are above 0.8. most of Train-STCS and Test-STCS are lower than 0.3. Therefore, it indicates that FreeTalk can achieve satisfactory privacy-preserving performance.
(2) Speech quality and utility: After adding the universal perturbation patch, protected audio maintains high speech quality and utility. From Table IV, we can know that most of MOS remain above 2.5. It should be noted that when the noise frame length is 120, the overall MOS is higher than that when the noise frame length is 30. We indicate that under the same noise level conditions, when the noise frame length is longer, the added noise amplitude is lower. This results in better audio quality and a higher MOS value. Moreover, we can see that the CER and WER always remain at 0. This indicates that the noise frame length has little impact on the speech utility. The experimental results indicate that the universal perturbation patch generated by our method has some influence on speech quality and negligible influence on speech utility.
(3) Others: The proposed identity-wise optimization protection method is not sensitive to noise level and the speech verification model. From Table IV, we can see that the fluctuations of STCMR and STCS on the training set and the test set are small with the gradual increase of noise level. This indicates that the privacy protection performance is stable. Moreover, the MOS shows a little decreasing trend, indicating that the protected speech quality gradually decreased with the increase of noise level. However, the CER and WER are almost stable, indicating that there is little impact on the utility of the protected audio. Besides, the experimental results remain stable on the different speech verification models, which indicates that our method has great robustness.
V-F Distortion Analysis
To evaluate the perceptual distortion of FreeTalk, we conduct a visual inspection of the protected audio in both temporal and frequential domains, as shown in Fig. 5.
In the temporal plots, we observe that the waveforms of FreeTalk-protected audios remain closely aligned with the original signal. The overall phase and temporal progression are unaffected, with only subtle amplitude perturbations being introduced. These perturbations manifest as slight upward or downward shifts in local peaks and troughs, yet the global waveform envelope is preserved. Such characteristics indicate that the prosodic rhythm, syllable timing, and energy contours of speech are maintained, which are critical for intelligibility and the perception of natural speech. From a perceptual standpoint, this ensures that listeners do not experience noticeable artifacts or discontinuities. In the frequential domain, the Mel Spectrograms reveal distinct behaviors for the two protection strategies. For sample-wise protection, the perturbations appear as scattered, localized intensity changes across both low-frequency and mid-frequency bands. These fine-grained variations differ across individual utterances, reflecting the stochastic and instance-dependent nature of this protection. Importantly, these modifications do not obscure the broader formant structures or harmonic patterns, thereby preserving phonetic clarity while injecting sufficient randomness to hinder model inversion or feature reconstruction. By contrast, identity-wise protection introduces perturbations that are more consistent across the entire spectro-temporal span. Rather than appearing as isolated fluctuations, these perturbations form smoother, frame-spanning adjustments that shift the energy distribution of the spectrum in a controlled manner. This uniformity encodes protection signals that are stable across all utterances from the same speaker, effectively obfuscating speaker identity. While the deviations are slightly stronger than in the sample-wise case, they remain constrained in magnitude such that the essential timbre and resonance characteristics of the original voice are preserved. Consequently, the speech continues to sound natural to human listeners, but automatic speaker recognition models face greater difficulty in associating the protected audio with the true identity.
Overall, both protection modes strike a balance between robustness and imperceptibility: sample-wise perturbations emphasize subtle, transient distortions, while identity-wise perturbations embed stable, identity-level shifts. Together, these demonstrate that FreeTalk achieves effective protection with low perceptual degradation.
VI Conclusion
In this paper, we firstly propose a sample-wise privacy protection method against speech synthesis attacks. Our method protects user’s voice characteristic by adding a frequency-domain perturbation. Then, to improve the transferability and robustness of our method, we also propose data augmentation strategy and noise smoothing mechanism to maintain natural speech quality. Finally, we also propose a novel identity-wise protection mechanism, which can support any length speech and generate a universal perturbation for a speaker. It should be noted that our method can perform well in a black-box setting. Extensive experimental results show the superior performance of FreeTalk on different models and datasets. The protected speech has satisfied privacy preservation performance, high speech quality and utility. In addition, the experimental results also demonstrate that the proposed FreeTalk has a fine generalization ability on different noise levels, different frequency intervals, and different speech verification models. The relevant experimental results also indicate that our method has better potential practicality for real-world scenarios.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] S. Liu, Y. Cao, D. Wang, X. Wu, X. Liu, and H. Meng, “Any-to-many voice conversion with location-relative sequence-to-sequence modeling,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 29, pp. 1717–1728, 2021.
- 2[2] Z.-Y. Sheng, Y. Ai, Y.-N. Chen, and Z.-H. Ling, “Face-driven zero-shot voice conversion with memory-based face-voice alignment,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 8443–8452.
- 3[3] Z. Yu, S. Zhai, and N. Zhang, “Antifake: Using adversarial audio to prevent unauthorized speech synthesis,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security , 2023, pp. 460–474.
- 4[4] Y. Deng, H. Tang, X. Zhang, J. Wang, N. Cheng, and J. Xiao, “Pmvc: Data augmentation-based prosody modeling for expressive voice conversion,” in Proceedings of the 31st ACM International Conference on Multimedia , 2023, pp. 184–192.
- 5[5] J. James, B. Balamurali, C. I. Watson, and B. Mac Donald, “Empathetic speech synthesis and testing for healthcare robots,” International Journal of Social Robotics , vol. 13, no. 8, pp. 2119–2137, 2021.
- 6[6] Y. Jia, R. J. Weiss, F. Biadsy, W. Macherey, M. Johnson, Z. Chen, and Y. Wu, “Direct speech-to-speech translation with a sequence-to-sequence model (2019),” ar Xiv preprint ar Xiv:1904.06037 , vol. 21, 1904.
- 7[7] J. Mertl, E. Žáčková, and B. Řepová, “Quality of life of patients after total laryngectomy: the struggle against stigmatization and social exclusion using speech synthesis,” Disability and Rehabilitation: Assistive Technology , vol. 13, no. 4, pp. 342–352, 2018.
- 8[8] S. Nakamura, “Overcoming the language barrier with speech translation technology,” Science & Technology Trends-Quarterly Review , vol. 31, 2009.