Pruning Self-Attention for Zero-Shot Multi-Speaker Text-to-Speech

TL;DR

This paper introduces a differentiable pruning method for transformer-based TTS models, enhancing their ability to generalize to unseen speakers in zero-shot multi-speaker speech synthesis.

Contribution

It proposes a novel differentiable sparse attention pruning technique that automatically learns optimal thresholds to improve out-of-domain generalization in TTS.

Findings

Improved voice quality in zero-shot multi-speaker TTS.

Enhanced speaker similarity with the proposed pruning method.

Effective reduction of redundant self-attention connections.

Abstract

For personalized speech generation, a neural text-to-speech (TTS) model must be successfully implemented with limited data from a target speaker. To this end, the baseline TTS model needs to be amply generalized to out-of-domain data (i.e., target speaker's speech). However, approaches to address this out-of-domain generalization problem in TTS have yet to be thoroughly studied. In this work, we propose an effective pruning method for a transformer known as sparse attention, to improve the TTS model's generalization abilities. In particular, we prune off redundant connections from self-attention layers whose attention weights are below the threshold. To flexibly determine the pruning strength for searching optimal degree of generalization, we also propose a new differentiable pruning method that allows the model to automatically learn the thresholds. Evaluations on zero-shot multi-speaker TTS verify the effectiveness of our method in terms of voice quality and speaker similarity.