Improved Conditional VRNNs for Video Prediction

TL;DR

This paper introduces an enhanced hierarchical variational autoencoder framework with more expressive latent distributions and higher capacity likelihood models to improve the accuracy and quality of video frame prediction.

Contribution

It proposes a novel hierarchical latent variable model that increases the expressiveness of the latent space for better future frame prediction in videos.

Findings

Outperforms current state-of-the-art models on multiple datasets.

Produces sharper and more accurate future video frames.

Demonstrates the effectiveness of hierarchical latent structures.

Abstract

Predicting future frames for a video sequence is a challenging generative modeling task. Promising approaches include probabilistic latent variable models such as the Variational Auto-Encoder. While VAEs can handle uncertainty and model multiple possible future outcomes, they have a tendency to produce blurry predictions. In this work we argue that this is a sign of underfitting. To address this issue, we propose to increase the expressiveness of the latent distributions and to use higher capacity likelihood models. Our approach relies on a hierarchy of latent variables, which defines a family of flexible prior and posterior distributions in order to better model the probability of future sequences. We validate our proposal through a series of ablation experiments and compare our approach to current state-of-the-art latent variable models. Our method performs favorably under several metrics in three different datasets.

Tables

Table 1. Table 1: Ablation - Likelihood We compare models with different number of recurrent layers for the likelihood network. We observe that the model performance increases monotonically as we add more ConvLSTMs. We further increase the size of the recurrent hidden states for the 6-ConvLSTM model (+ higher capacity variant), also leading to a better data fit. These results suggest that current video prediction models might underfit the data because of reduced likelihood capacity.

Model	Parameters	Train/Test ELBO($\downarrow$)
1-ConvLSTM	62.22M	3237/3826
3-ConvLSTM	86.64M	1948/2355
6-ConvLSTM	93.81M	1279.21/1731.31
+ higher capacity	194.15M	1113.31/1589.72

Table 2. Table 2: Ablation - Hierarchy of Latents Top half: We compare a VRNN baseline with a single level of latents with no spatial topology (1), a model with two levels of latents at resolutions 1x1 and 8x8 (1-8), our full model with three levels of latents at 1x1, 8x8 and 32x32 (1-8-32), and a model with 4 levels of latents (1-8-16-32). Adding more levels of latents leads to a better fit, with reduced ELBOs. However, adding extra levels of latents without increasing the spatial resolution reduces the performance of the model due to the difficulties in training hierarchical latent variable models. Bottom half: To highlight the difficulties in training hierarchies of latents, we investigate the effects of using beta warmup (BW)  [ 31 ] and having a dense connectivity (Dense) between latents when training the 1-8-32 model. Without these techniques the hierarchy of latents does not bring any benefit compared to the VRNN with 1 level of latent.

Model	Parameters	Train/Test ELBO ($\downarrow$)
1	166.55M	1141.85/1536.93
1-8	220.60M	989.39/1313.02
1-8-32	230.74M	883.10/1162.24
1-8-16-32	245.19M	956.63/1256.22
Naive Training	224.18M	1127.33/1440.58
BW	224.18M	1101.39/1440.62
Dense	230.74M	1182.60/1547.05
BW and Dense	230.74M	883.10/1162.24

Table 3. Table 3: Stochastic Moving MNIST. We compute the FVD metric between samples from different models and test sequences as well as the average LPIPS and SSIM of the best sample for each test sequence. Our models outperform the SVG-LP baseline on all metrics by a significant margin. While our model with hierarchical latent variables obtains a better FVD score, both variants obtain comparable results in this relatively simple dataset.

Model	FVD ($\downarrow$)	LPIPS ($\downarrow$)	SSIM ($\uparrow$)
SVG-LP  [4]	90.81	0.153 $\pm$ 0.03	0.668 $\pm$ 0.04
\hdashlineOurs w/o hier	63.81	0.102 $\pm$ 0.04	0.763 $\pm$ 0.09
Ours w/ hier	57.17	0.103 $\pm$ 0.03	0.760 $\pm$ 0.08

Equations

$$p(\mathbf{x},\mathbf{z}|\mathbf{c})=\prod _{t=1}^{T}p(x_t|{\mathbf{z}}_{\le \mathbf{t}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c})p(z_t|{\mathbf{z}}_{<\mathbf{t}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c}).$$

$$\begin{gathered} \log p(\mathbf{x}|\mathbf{c})\ge \sum _{t=1}^T{\mathbb{E}}_{q({\mathbf{z}}_{\le \mathbf{t}}|{\mathbf{x}}_{\le \mathbf{t}},\mathbf{c})}\log p(x_t|{\mathbf{z}}_{\le \mathbf{t}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c}) \\ -D_{KL}(q(z_t|{\mathbf{z}}_{<\mathbf{t}},{\mathbf{x}}_{\le \mathbf{t}},\mathbf{c})||p(z_t|{\mathbf{z}}_{<\mathbf{t}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c})) \end{gathered}$$

$$p({\mathbf{z}}_{\mathbf{t}}|{\mathbf{z}}_{<\mathbf{t}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c})=\prod _{l=1}^{L}p(z_t^l|{\mathbf{z}}_{\mathbf{t}}^{<\mathbf{l}},{\mathbf{z}}_{<\mathbf{t}}^{\mathbf{l}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c}).$$

$$q({\mathbf{z}}_{\mathbf{t}}|{\mathbf{z}}_{<\mathbf{t}},{\mathbf{x}}_{\le \mathbf{t}},\mathbf{c})=\prod _{l=1}^{L}q(z_t^l|{\mathbf{z}}_{\mathbf{t}}^{<\mathbf{l}},{\mathbf{z}}_{<\mathbf{t}}^{\mathbf{l}},{\mathbf{x}}_{\le \mathbf{t}},\mathbf{c}).$$

$$\begin{gathered} \log p(\mathbf{x}|\mathbf{c})\ge \sum _{t=1}^T[{\mathbb{E}}_{q({\mathbf{z}}_{\mathbf{t}}|{\mathbf{z}}_{<\mathbf{t}},{\mathbf{x}}_{\le \mathbf{t}},\mathbf{c})}\log p(x_t|{\mathbf{z}}_{\le \mathbf{t}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c}) \\ -\sum _{l=1}^L{D}_{KL}(q(z_t^l|{\mathbf{z}}_{\mathbf{t}}^{<\mathbf{l}},{\mathbf{z}}_{<\mathbf{t}}^{\mathbf{l}},{\mathbf{x}}_{\le \mathbf{t}},\mathbf{c})||p(z_t^l|{\mathbf{z}}_{\mathbf{t}}^{<\mathbf{l}},{\mathbf{z}}_{<\mathbf{t}}^{\mathbf{l}},{\mathbf{x}}_{<\mathbf{t}},\mathbf{c}))]. \end{gathered}$$

Full text

1 Introduction

††Correspondence to lluis.enric.castrejon.subira@umontreal.ca

We investigate the task of video prediction, a particular instantiation of self-supervision [6, 8] where generative models learn to predict future frames in a video. Training such models does not require any annotated data, yet the models need to capture a notion of the complex dynamics of real-world phenomena (such as physical interactions) to generate coherent sequences.

Uncertainty is an inherent difficulty associated with video prediction, as many future outcomes are plausible for a given sequence of observations [1, 4]. Predictions from deterministic models rapidly degrade over time as uncertainty grows, converging to an average of the possible future outcomes [32]. To address this issue, probabilistic latent variable models such as Variational Auto-Encoders (VAEs) [18, 29], and more specifically Variational Recurrent Neural Networks (VRNNs) [2], have been proposed for video prediction [1, 4]. These models define a prior distribution over a set of latent variables, allowing different samples from these latents to capture multiple outcomes.

It has been empirically observed that VAE and VRNN-based models produce blurry predictions [20, 21]. This tendency is usually attributed to the use of a similarity metric in pixel space [20, 24] such as Mean Squared Error (corresponding to a log-likelihood loss under a fully factorized Gaussian distribution). This has lead to alternative models such as VAE-GAN [20, 21], which extends the traditional VAE objective with an adversarial loss in order to obtain more visually compelling generations.

In addition, the lack of expressive latent distributions has been shown to lead to poor model fitting [12]. Training VAEs involves defining an approximate posterior distribution over the latent variables which models their probability after the generated data has been observed. If the approximate posterior is too constrained, it will not be able to match the true posterior distribution and this will prevent the model from accurately fitting the training data. On the other hand, the prior distribution over the latent variables can be interpreted as a model of uncertainty.

The decoder or likelihood network needs to transform latent samples into data observations covering all plausible outcomes. Given a simple prior, this transformation can be very complex and require high capacity networks. We hypothesize that the reduced expressiveness of current VRNN models is limiting the quality of their predictions and investigate two main directions to improve video prediction models. First, we propose to scale the capacity of the likelihood network. We empirically demonstrate that by using a high capacity decoder we can ease the latent modeling problem and better fit the data.

Second, we introduce more flexible posterior and prior distributions [30]. Current video prediction models usually rely on one shallow level of latent variables and the prior and approximate posterior are parameterized using diagonal Gaussian distributions [1]. We extend the VRNN formulation by proposing a hierarchical variant that uses multiple levels of latents per timestep.

Models leveraging a hierarchy of latents are known to be hard to optimize as they are required to backpropagate through a stack of stochastic latent variables, usually resulting in models that only make use of a small subset of the latents [18, 23, 30]. We mitigate this problem by using a warmup regime for the KL loss [31] and a dense connectivity pattern [13, 22] between the input and latent variables. Specifically, each stochastic latent variable is connected to the input and to all subsequent stochastic levels in the hierarchy. Our empirical findings confirm that only with these techniques our model is able to take advantage of different layers in a latent hierarchy.

We validate our hierarchical VRNN in three datasets with varying levels of future uncertainty and realism: Stochastic Moving MNIST  [4], the BAIR Push Dataset  [7] and Cityscapes  [3]. When compared to current state of the art models [4, 21], our approach performs favorably under several metrics. In particular for the BAIR Push Dataset, our hierarchical-VRNN shows an improvement of $44\%$ in Video Fréchet Distance (FVD) [34] and $9.8\%$ in term of LPIPS score [41] over SVG-LP [4], the previous best VAE-based model. It also achieves a similar FVD than the SAVP VAE-GAN model [21], while showing a $11.2\%$ improvement in terms of LPIPS over this baseline.

2 Related Work

Initial video prediction approaches relied on deterministic models. Ranzato et al. [27] divided frames into patches and predicted their evolution in time given previous neighboring patches. In [32] Srivastava et al. used LSTM networks on pretrained image embeddings to predict the future. Similarly, Oh et al. [25] used LSTMs on CNN representations to predict frames in Atari games when given the player actions.

ConvLSTMs [40] adapt the LSTM equations to spatial feature maps by replacing matrix multiplications with convolutions. They were originally used for precipitation nowcasting and are commonly used for video prediction.

Other works have proposed to disentangle the motion and context of the frames to generate [35, 33, 5]. They assume that a scene can be decomposed as multiple objects, which allows them to use a fixed representation for the background. Our approach does not follow this modeling assumption and instead tries to capture the uncertainty in the future.

Autoregressive models [15, 28] approximate the full joint data distribution $p(x_{1},x_{2},...,x_{N})$ over pixels, which allows them to capture complex pixel dependencies but at the expense of making their inference mechanism slow and not scalable to high resolutions. Latent variable models using GANs [9] were proposed in  [37, 36, 33]. Training pure GAN video models is still an open research direction: training is unstable and most models require auxiliary losses.

A successful approach so far has been based on VAE [18, 29]/VRNN [2] models. SV2P [1] proposed to capture sequence uncertainty in a single set of latent variables kept fixed for each predicted sequence. SVG [4] adopted the VRNN formulation [2], introducing per-step latent variables (SVG-FP) and a variant with a learned prior (SVG-LP), which makes the prior at a certain timestep a function of previous frames. In recent work, SAVP [21] proposed to use the VAE-GAN [20] framework for video, a hybrid model that offers a trade-off between VAEs and GANs. Our model extends the VRNN formulation by introducing a hierarchy of latents to better approximate the data likelihood.

There are multiple works addressing hierarchical VAEs for non-sequential data [26, 23, 31, 17]. While hierarchical VAEs can model more flexible latent distributions, training them is usually difficult due to the multiple layers of conditional latents [30]. Ladder Variational Autoencoders [31] proposed a series of techniques to partially alleviate this issue. IAF [17] used a similar architecture to Ladder VAEs and extended it with a novel normalizing flow. Recent work [22] has trained very deep hierarchical models that produce visually compelling samples. We extend hierarchical latent variable models to sequential data and apply them to video prediction. Concurrent work  [19] has proposed a fully invertible model for video.

3 Preliminaries

We follow previous work in video prediction [4]. Given $D$ context frames $\mathbf{c}=(c_{1},c_{2},...,c_{D})$ and the $T$ following future frames $\mathbf{x}=(x_{1},x_{2},...,x_{T})$, our goal is to learn a generative model that maximizes the probability $p(\mathbf{x}|\mathbf{c})$.

VRNN follows the VAE formalism and introduces a set of latent variables $\mathbf{z}=(z_{1},z_{2},...,z_{T})$ to capture the variations in the observed variables $\mathbf{x}$ at each timestep $t$. It defines a likelihood model $p(\mathbf{x}|\mathbf{z},\mathbf{c})=\prod_{t=1}^{T}p(x_{t}|\mathbf{z_{\leq t}},\mathbf{x_{<t}},\mathbf{c})$ and a prior distribution $p(\mathbf{z}|\mathbf{c})=\prod_{t=1}^{T}p(z_{t}|\mathbf{z_{<t}},\mathbf{x_{<t}},\mathbf{c})$ which are parametrized in an autoregressive manner; i.e. at each timestep observed and latent variables are conditioned on the past latent samples and observed frames. VRNN therefore uses a learned prior [2, 4]. Taking into account the temporal structure of the data, the probability $p(\mathbf{x},\mathbf{z}|\mathbf{c})$ is factorized as

[TABLE]

Computing $p(\mathbf{x}|\mathbf{c})$ requires marginalizing over the latent variables $\mathbf{z}$, which is computationaly intractable. Instead, VRNN relies on Variation Inference [14] and defines an amortized approximate posterior $q(\mathbf{z}|\mathbf{x},\mathbf{c})=\prod_{t=1}^{T}q(z_{t}|\mathbf{z_{<t}},\mathbf{x_{\leq t}},\mathbf{c})$ that approximates the true posterior distribution $p(\mathbf{z}|\mathbf{x},\mathbf{c})$. We then can derive the evidence lower bound (ELBO), a lower bound to the marginal log-likelihood $p(\mathbf{x}|\mathbf{c})$:

[TABLE]

VRNN can be optimized to fit the training data by maximizing the ELBO using stochastic backpropagation and the reparameterization trick [18, 29].

4 Hierarchical VRNN

We now introduce a hierarchical version of the VRNN model. At each timestep, we consider $L$ levels of latents variables $\mathbf{z_{t}}=({z^{1}_{t}},...,{z^{L}_{t}})$. We then further factorize the latent prior as

[TABLE]

The sampling process of the latent variable $z_{t}^{l}$ now depends on the latent variables from previous time steps $\mathbf{z_{<t}^{l}}$ for that level and on the latent variables of the previous levels at the current timestep $\mathbf{z_{t}^{<l}}$. Similarly, we can write the approximate posterior as:

[TABLE]

Using eq. 3 and eq.4, we can rewrite the ELBO as

[TABLE]

Refer to the Appendix for a full derivation of the ELBO.

4.1 Dense Latent Connectivity

Training a hierarchy of latent variables is known to be challenging as it requires to backpropagate through multiple stochastic layers. Usually this results in models that only use one specific level of the hierarchy [18, 23, 30]. To ease the optimization we use a dense connectivity pattern between latent levels both for the prior and the approximate posterior, following [13, 22].

Fig 2 illustrates the dense connection of the learned prior (refer to the Appendix for the approximate posterior). For each latent level, the prior and posterior are implemented using recurrent neural networks which take as input a deterministic transformation of $x_{t-1}$ (red arrows in Fig 2), and to all the latent variables from the previous levels (green arrows in in Fig 2). In addition, each latent variable has a direct connection to the output variables $x_{t}$.

4.2 Model Parametrization

We now describe an instantiation of the VRNN model that we will use in the experiments, illustrated in Fig. 3. First we compute features for each context frame and use them to initialize the hidden state of the prior/posterior/decoder networks, all of which have recurrent components. At a given timestep $t$, the model takes as input the latent variable samples $\mathbf{z_{t}}=(z_{t}^{0},...,z_{t}^{L})$ with the embedding of the previously generated frame $x_{t-1}$ and outputs the next frame $\hat{x}_{t}$. During training we draw latent samples from the approximate posterior distribution $q(\mathbf{z_{t}}|\mathbf{z_{<t}},\mathbf{x_{\leq t}},\mathbf{c})$ and maximize the ELBO. To generate unseen sequences, we sample $\mathbf{z_{t}}$ from the learned prior $p(\mathbf{z_{t}}|\mathbf{z_{<t}},\mathbf{x_{<t}},\mathbf{c})$. Note that since we have multiple levels of conditional latents we use ancestral sampling to generate $\mathbf{z_{t}}$, i.e. we first sample from the top level of the hierarchy and we then sequentially sample the lower levels conditioning on the sampled values of the previous layers in the hierarchy.

Frame Encoder We use a 2D CNN with ResNet [11] blocks and max-pooling layers to represent input frames.

Prior/Approximate Posterior We parametrize both the prior and the posterior as a hierarchy of diagonal Normal distributions $\mathcal{N}(\mu,\sigma)$, where the parameters $\mu$ and $\sigma$ are recurrent functions of samples from i) previous levels in the hierarchy and ii) the frame encoder features. Each level in the hierarchy operates at a different spatial resolution, with the top level features operating at a 1x1 resolution, i.e. not having a spatial topology. At a given timestep $t$, the parameters for a specific latent level $z^{l}_{t}$ are given by a ConvLSTM that consumes i) a previous hidden state, ii) samples from the previous levels in the hierarchy $z_{t}^{<l}$, iii) the feature map of a frame with the same spatial resolution as the ConvLSTM. For the prior network, the input frame embedding corresponds to the previously generated frame $x_{t-1}$, while for the posterior the input comes from the frame to generate $x_{t}$.

Likelihood/Frame Decoder At each timestep $t$, the decoder takes a representation of the previously generated frame $x_{t-1}$ and the samples $\mathbf{z_{t}}=(z_{t}^{1},...,z_{t}^{L})$ and generates $x_{t}$ according to $p(x_{t}|\mathbf{z_{t}},\mathbf{x_{<t}},\mathbf{c})$. The decoder consists of ConvLSTMs interleaved with transposed convolutions that upscale the feature maps back to the input resolution.

Initial State The initial states of our prior, posterior and decoder/likelihood models are functions of the context. We use small CNNs to initialize each of the ConvLSTMs layers used in the VAE components.

5 Experiments

All our models are trained with Adam [16] and a batch size of $b=128$ on Nvidia DGX-1s. We use a learning rate warmup [10] starting with an initial learning rate $\lambda$ = 2e-5 that is linearly increased at each timestep until reaching $\lambda$ = 1.6e-4 in 5 epochs. We use $\beta_{1}$ = 0.5 and $\beta_{2}$ = 0.9 and weight decay $\delta$ = 1e-4. We train the autoregressive components of our models using teacher forcing [39].

Our models are also trained using beta warmup [31], which consists in gradually increasing the weight of the KL divergence in the ELBO, turning the model from an unregularized Autoencoder into a VAE progressively. VAEs trained with beta warmup usually encode more information in the latent variables. Refer to the Appendix for a complete description of our models.

5.1 Ablation Study

We first investigate the importance of each VRNN component, namely the likelihood, the prior and the posterior. We focus on the BAIR Push dataset [7] with 64x64 color sequences of a robotic arm interacting with children toys in a sandbox. Similarly to previous works [21], we use trajectories 256 to 511 as our test set and the rest for training, resulting in the 43264 train and 256 test sequences. At training we randomly subsample 12 frames from each train sequence, use the first 2 frames as the context, and learn to predict the remaining 10 frames. To evaluate the different model variations, we report the training objective (ELBO) obtained for the training set and the test set.

5.1.1 Scaling the Likelihood Model

We assess the importance of the likelihood model $p(x_{t}|\mathbf{z_{\leq t}},\mathbf{x_{<t}},\mathbf{c})$. For this purpose, we build a VRNN with a single level of latent variables and modify the number of ConvLSTM layers in the decoder. Our aim is to investigate whether increasing the capacity of the mapping from latent to the observations results in better predictions.

In this experiment, our baseline likelihood model has one LSTM at 1x1 spatial resolution. We then gradually replace convolutional layers in the decoder with ConvLSTM layers, which increases the amount of information that can be carried from previous timesteps and, by extension, also increases the overall likelihood model capacity. We compare to a model with 3 ConvLSTM layers at resolutions 1x1, 4x4 and 8x8 and a model with 6 ConvLSTM layers at 1x1, 4x4, 8x8, 16x16, 32x32 and 64x64. Additionally, we also increase the size of the ConvLSTM layers for the model with 6 layers as another way of adding capacity.

Results can be found in Fig 1. We observe that, as a general trend, both the training and test ELBO decrease as we increase the model capacity, which suggests that current video prediction models might operate in an underfitting regime and benefit from higher capacity decoders.

5.1.2 More Flexible Prior and Posterior

We now investigate the importance of having more flexible prior and approximate posterior distributions and augment the 6-ConvLSTM VRNN model with a hierarchy of latent variables. For all models, we fix the frame encoder and likelihood model111To add the multiple levels of latents in the decoder we need to modify the likelihood network and slightly increase the number of parameters. However, most ($>85\%$) of the added capacity when adding a new level of latents goes towards the prior and posterior networks. and change the networks that estimate the learned prior $p(\mathbf{\mathbf{z_{t}}}|\mathbf{z_{<t}},\mathbf{x_{<t}},\mathbf{c})$, and the approximate posterior $q(\mathbf{\mathbf{z_{t}}}|\mathbf{z_{<t}},\mathbf{x_{\leq t}},\mathbf{c})$ over the latent variables. All these models use a dense connectivity pattern and beta warmup.

We compare a VRNN baseline with a single level of latents with no spatial topology, with a model with two levels of latents at resolutions 1x1 and 8x8 (1-8), three levels of latents at 1x1, 8x8 and 32x32 (1-8-32), and four levels of latents (1-8-16-32) in the top half of Table 2. All models are trained with beta warmup and dense latent connectivity. We observe that in general adding more levels of latents with higher resolution reduces the train and test ELBOs, supporting the hypothesis that a more flexible prior and posterior leads to a better data fit. However, we observe diminishing returns past 3 levels, as our 1-8-16-32 model does not outperform the 3 layers model. We attribute this to the difficulties in training deep hierarchies of latents, which remains a challenging optimization problem.

To further highlight the difficulties in training hierarchies of latents, we investigate the importance of using beta warmup [31] and having a dense connectivity between latents. The results of this experiment can be found in the bottom half of Table 2. We observe that these techniques are required to make our 1-8-32 model make use of the hierarchy of latents and improve upon the single level model.

This is analyzed in more detail in Fig 4, where we visualize the KL between the prior and the posterior distributions for the test sequences of the BAIR Push dataset for the 1-8-32 model and the variant without warmup or dense connectivity (Naive training). We consider a channel to be active if its average KL is higher than 0.01 following [22], and consider that a unit with a KL higher than 0.15 is maximally activated. We observe that without these techniques the model only uses a few latents of the top level in the hierarchy. However, when using beta warmup and a dense connectivity most of the latents are active across levels.

5.2 Comparisons to Previous Approaches

Next, we compare our single latent level VRNN (Ours w/o Hier), and our hierarchical VRNN with 3 levels of latents (Ours w/ Hier) to previous video approaches on Stochastic Moving MNIST  [4], BAIR Push [7] and the Cityscapes  [3] datasets.

5.2.1 Evaluation and Metrics

Defining evaluation metrics for video prediction is an open research question. In general we want models to predict sequences that are plausible, look realistic and cover all possible outcomes. Unfortunately, we are not aware of any metric that reflects all these aspects.

To measure coverage and plausibility we adopt the evaluation protocol proposed in [4, 21]. For each ground truth test sequence, we sample $N$ random predictions from the model which are conditioned on the test sequence initial frames. Then we find the sample that best matches the ground truth sequence according to a given metric and report that metric value. Some common metric choices are Mean-Square Error (MSE), Structural Similarity (SSIM) [38] or Peak Signal-to-Noise Ratio (PSNR). In practice, these metrics have been shown to not correlate well with human judgement as they tend to prefer blurry predictions over sharper but imperfect generations [41, 21, 34]. LPIPS [41], on the other hand, is a perceptual metric that employs CNN features and has better correlation to human judgment. For this evaluation we choose to produce $N=$ 100 samples following previous work and use SSIM and LPIPS as metrics. We have empirically observed that using 100 samples the results stay fairly consistent across different samplings. We report the metric average over the test set.

Additionally, we also use the recently proposed Fréchet Video Distance (FVD), which measures sample realism. FVD uses features from a 3D CNN and has also been shown to correlate well with human perception [34]. FVD compares populations of samples to assert whether they were both generated by the same distribution (it does not compare pairs of ground truth/generated frames directly). We form the ground truth population by using all the test sequences with their context. For the predicted population we randomly sample one video out of the $N$ generated for each test sequence. We repeat this process 5 times and report the mean of the FVD scores obtained, which stay fairly stable across samplings.

5.2.2 Stochastic Moving MNIST

Stochastic Moving MNIST is a synthetic dataset proposed in [4] which consists of black and white sequences of MNIST digits moving over a black background and bouncing off the frame borders. As opposed to the original Moving MNIST dataset [32] with completely deterministic motion, Stochastic Moving MNIST has uncertain digit trajectories - the digits bounce off the border with a random new trajectory. We train two variants of our model and compare to the SVG-LP baseline [4], for which we use a pretrained model from the official codebase. All models are trained using 5 frames of context and 10 future frames to predict. To evaluate the models, we follow the procedure in [4] described in section 5.2.1.

We report the results of the experiment in Table 3. We observe that both versions of our model (with/out the latent hierarchy) outperform the SVG-LP baseline by a significant margin on all metrics. Note that LPIPS and FVD might not be suited to this dataset as they use features from CNNs trained on real world images, but we report them for completeness. Visually, our samples (found in the Appendix) reproduce the digits more faithfully with reduced degradation over time. There are small differences between the two versions of our model, suggesting that the extra expressiveness of the hierarchical model is not necessary in this synthetic dataset.

5.2.3 BAIR Push

We compare our VRNN models to SVG-LP [4] and SAVP [21]. We use their official implementations and pretrained models to reproduce their results. We use the experimental setup of previous works  [4, 21], using 2 context frames and generating 28 frames.

Results can be found on Fig 6. When the robotic arm is interacting with an object, SVG-LP tends to generate blurry predictions characterized by a high FVD score. SAVP exhibits a lower FVD as it produces more realistically looking predictions. However, SAVP does not have a better coverage of the ground truth sequences compared to SVG-LP as measured by LPIPS and SSIM. By inspecting the SAVP samples we notice that the SAVP generations tend to be sharper but sometimes they exhibit temporal inconsistencies or implausible interactions (see Fig 5). Our w/o Hier VRNN models obtain better scores than SVG-LP, the previous best VAE-type model. This supports the importance of having a high-capacity likelihood model. In addition, our hierarchical VRNN further improves both the FVD and LPIPS metrics, suggesting that the hierarchy of latents helps modeling the data In particular, our hierarchical VRNN shows an improvement of $44\%$ in terms of FVD and $9.8\%$ in terms of LPIPS over SVG-LP, the previous best VAE-based model. It also achieves a similar FVD than the SAVP GAN-VAE model, while outperforming it in terms of LPIPS by $11.2\%$.

5.2.4 Cityscapes

The Cityscapes dataset contains sequences recorded from a car driving around multiple cities under different conditions. Cityscapes is a challenging dataset - while contiguous frames are locally similar, uncertainty grows significantly with time. Compared to previous datasets, the backgrounds in Cityscapes do not stay constant across time.

We consider sequences with 30 frames from the training set cities for a total of 1877 train sequences and randomly select 256 test sequences. We use 2 context and 10 prediction frames to train the models. At test time we predict 28 frames following the BAIR Push experimental protocol. We preprocess the videos by taking a 1024x1024 center crop of the original sequences and resizing them to 128x128 pixels. For evaluating the models we use the standard setup where we generate 100 samples per test sequence and report FVD, SSIM and LPIPS metrics. Since none of the baselines from previous experiments are trained on Cityscapes, we use the official SVG implementation (that defines models with 128x128 inputs) and train a SVG-LP model. We train all models for 100 epochs.

Results for this experiment can be found in Fig. 7. SVG-LP has trouble modelling motion in the dataset, usually predicting a static image similar to the last context frame. In contrast, our model without a hierarchy of latents is able to model the changing scene. When adding hierarchical latents our model is able to capture more fine-grained details, and as a result, it produces more visually appealing samples with a boost in all metrics. We note that the SSIM scores for SVG-LP match those of our models at later timesteps in the prediction, however this does not translate to better samples as can be seen in Fig. 5 or in the Appendix. This further indicates that SSIM might not be suitable to evaluate video prediction models.

6 Conclusions

We propose a hierarchical VRNN for video prediction that features an improved likelihood model and a hierarchy of latent variables. Our approach compares favorably to current state of the art models in terms of the Fréchet Video Distance, LPIPS and SSIM metrics, producing visually appealing and coherent samples. Our results demonstrate that current video prediction models benefit from increased capacity, opening the door to further gains with bigger and more flexible generative models.