Learning Stochastic Recurrent Networks

TL;DR

This paper introduces Stochastic Recurrent Networks (STORNs), which incorporate latent variables into RNNs using variational inference, enabling structured probabilistic modeling and improved training for sequential data.

Contribution

The paper presents a novel stochastic RNN model that can be trained efficiently with stochastic gradients and supports complex conditional distributions.

Findings

Effective on polyphonic music datasets

Outperforms deterministic RNNs in modeling complex sequences

Provides reliable marginal likelihood estimation

Abstract

Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a reliable estimator of the marginal likelihood and iv) is a generalisation of deterministic recurrent neural networks. We evaluate the method on four polyphonic musical data sets and motion capture data.