Benchmarking Generative Latent Variable Models for Speech

TL;DR

This paper benchmarks various stochastic and deterministic latent variable models for speech, evaluating their likelihoods and representation quality, and introduces adaptations of video models to improve speech generation performance.

Contribution

It provides a comprehensive benchmark for temporal LVMs in speech, compares them with deterministic models, and adapts the Clockwork VAE for speech, showing improved results.

Findings

Clockwork VAE outperforms previous LVMs in speech.

Hierarchical latent variables help close the gap to deterministic models.

Likelihood metrics are effectively used to evaluate speech models.

Abstract

Stochastic latent variable models (LVMs) achieve state-of-the-art performance on natural image generation but are still inferior to deterministic models on speech. In this paper, we develop a speech benchmark of popular temporal LVMs and compare them against state-of-the-art deterministic models. We report the likelihood, which is a much used metric in the image domain, but rarely, or incomparably, reported for speech models. To assess the quality of the learned representations, we also compare their usefulness for phoneme recognition. Finally, we adapt the Clockwork VAE, a state-of-the-art temporal LVM for video generation, to the speech domain. Despite being autoregressive only in latent space, we find that the Clockwork VAE can outperform previous LVMs and reduce the gap to deterministic models by using a hierarchy of latent variables.