Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech

TL;DR

This paper introduces a novel machine unlearning framework for zero-shot text-to-speech systems, enabling models to forget specific speaker identities while preserving overall speech quality, addressing privacy concerns.

Contribution

It proposes the first unlearning methods for ZS-TTS, including Teacher-Guided Unlearning and a new evaluation metric, speaker-ZRF, to effectively forget specific voices.

Findings

TGU prevents voice replication of forgotten speakers.

Models retain high speech quality for other speakers.

The new metric accurately measures unlearning effectiveness.

Abstract

The rapid advancement of Zero-Shot Text-to-Speech (ZS-TTS) technology has enabled high-fidelity voice synthesis from minimal audio cues, raising significant privacy and ethical concerns. Despite the threats to voice privacy, research to selectively remove the knowledge to replicate unwanted individual voices from pre-trained model parameters has not been explored. In this paper, we address the new challenge of speaker identity unlearning for ZS-TTS systems. To meet this goal, we propose the first machine unlearning frameworks for ZS-TTS, especially Teacher-Guided Unlearning (TGU), designed to ensure the model forgets designated speaker identities while retaining its ability to generate accurate speech for other speakers. Our proposed methods incorporate randomness to prevent consistent replication of forget speakers' voices, assuring unlearned identities remain untraceable. Additionally, we propose a new evaluation metric, speaker-Zero Retrain Forgetting (spk-ZRF). This assesses the model's ability to disregard prompts associated with forgotten speakers, effectively neutralizing its knowledge of these voices. The experiments conducted on the state-of-the-art model demonstrate that TGU prevents the model from replicating forget speakers' voices while maintaining high quality for other speakers. The demo is available at https://speechunlearn.github.io/