Improving Sequential Latent Variable Models with Autoregressive Flows

TL;DR

This paper introduces autoregressive normalizing flows to enhance sequence modeling by reducing temporal correlations, resulting in better likelihood performance and generalization in video data.

Contribution

It presents a novel method combining autoregressive flows with sequence models, improving their ability to capture complex dynamics and decorrelate temporal data.

Findings

Improved log-likelihood on benchmark video datasets

Enhanced decorrelation and generalization properties

Effective as both standalone and integrated models

Abstract

We propose an approach for improving sequence modeling based on autoregressive normalizing flows. Each autoregressive transform, acting across time, serves as a moving frame of reference, removing temporal correlations, and simplifying the modeling of higher-level dynamics. This technique provides a simple, general-purpose method for improving sequence modeling, with connections to existing and classical techniques. We demonstrate the proposed approach both with standalone flow-based models and as a component within sequential latent variable models. Results are presented on three benchmark video datasets, where autoregressive flow-based dynamics improve log-likelihood performance over baseline models. Finally, we illustrate the decorrelation and improved generalization properties of using flow-based dynamics.