VQalAttent: a Transparent Speech Generation Pipeline based on Transformer-learned VQ-VAE Latent Space

TL;DR

VQalAttent is a lightweight, interpretable speech generation model that uses VQ-VAE and transformer architectures to produce high-quality, controllable fake speech efficiently, aiding understanding and development of advanced synthesis systems.

Contribution

The paper introduces VQalAttent, a novel, transparent pipeline combining VQ-VAE and transformer models for efficient, interpretable speech synthesis with limited computational resources.

Findings

Generates intelligible speech from discrete latent representations.

Provides insights into the relationship between latent space and speech quality.

Achieves high-quality speech synthesis with modular, transparent architecture.

Abstract

Generating high-quality speech efficiently remains a key challenge for generative models in speech synthesis. This paper introduces VQalAttent, a lightweight model designed to generate fake speech with tunable performance and interpretability. Leveraging the AudioMNIST dataset, consisting of human utterances of decimal digits (0-9), our method employs a two-step architecture: first, a scalable vector quantized autoencoder (VQ-VAE) that compresses audio spectrograms into discrete latent representations, and second, a decoder-only transformer that learns the probability model of these latents. Trained transformer generates similar latent sequences, convertible to audio spectrograms by the VQ-VAE decoder, from which we generate fake utterances. Interpreting statistical and perceptual quality of the fakes, depending on the dimension and the extrinsic information of the latent space, enables guided improvements in larger, commercial generative models. As a valuable tool for understanding and refining audio synthesis, our results demonstrate VQalAttent's capacity to generate intelligible speech samples with limited computational resources, while the modularity and transparency of the training pipeline helps easily correlate the analytics with modular modifications, hence providing insights for the more complex models.