Self-Attention Linguistic-Acoustic Decoder

TL;DR

This paper introduces a transformer-based linguistic-acoustic decoder for text-to-speech conversion that maintains low distortion while significantly improving inference speed, making it suitable for resource-constrained devices.

Contribution

The work presents a non-recurrent transformer decoder for speech synthesis that achieves comparable distortion to RNNs but with much faster inference times.

Findings

Competitive distortion levels with recurrent models

Over an order of magnitude faster inference

Suitable for deployment on resource-limited devices

Abstract

The conversion from text to speech relies on the accurate mapping from linguistic to acoustic symbol sequences, for which current practice employs recurrent statistical models like recurrent neural networks. Despite the good performance of such models (in terms of low distortion in the generated speech), their recursive structure tends to make them slow to train and to sample from. In this work, we try to overcome the limitations of recursive structure by using a module based on the transformer decoder network, designed without recurrent connections but emulating them with attention and positioning codes. Our results show that the proposed decoder network is competitive in terms of distortion when compared to a recurrent baseline, whilst being significantly faster in terms of CPU inference time. On average, it increases Mel cepstral distortion between 0.1 and 0.3 dB, but it is over an order of magnitude faster on average. Fast inference is important for the deployment of speech synthesis systems on devices with restricted resources, like mobile phones or embedded systems, where speaking virtual assistants are gaining importance.