Re-examination of the Role of Latent Variables in Sequence Modeling

TL;DR

This paper re-examines the role of latent variables in sequence modeling, revealing that standard recurrent models with auto-regressive outputs outperform stochastic models when fairness is ensured, challenging previous assumptions about their superiority.

Contribution

The study uncovers that previous evaluations favored stochastic models due to unfair restrictions, and demonstrates that standard recurrent models perform better when intra-step correlation is properly utilized.

Findings

Standard recurrent models outperform stochastic models under fair conditions.

Removing output distribution restrictions enables all models to leverage intra-step correlation.

Auto-regressive recurrent models achieve state-of-the-art results on speech datasets.

Abstract

With latent variables, stochastic recurrent models have achieved state-of-the-art performance in modeling sound-wave sequence. However, opposite results are also observed in other domains, where standard recurrent networks often outperform stochastic models. To better understand this discrepancy, we re-examine the roles of latent variables in stochastic recurrent models for speech density estimation. Our analysis reveals that under the restriction of fully factorized output distribution in previous evaluations, the stochastic models were implicitly leveraging intra-step correlation but the standard recurrent baselines were prohibited to do so, resulting in an unfair comparison. To correct the unfairness, we remove such restriction in our re-examination, where all the models can explicitly leverage intra-step correlation with an auto-regressive structure. Over a diverse set of sequential data, including human speech, MIDI music, handwriting trajectory and frame-permuted speech, our results show that stochastic recurrent models fail to exhibit any practical advantage despite the claimed theoretical superiority. In contrast, standard recurrent models equipped with an auto-regressive output distribution consistently perform better, significantly advancing the state-of-the-art results on three speech datasets.