Traversing Latent Space using Decision Ferns

TL;DR

This paper introduces a novel framework for controlled exploration of latent spaces using a Variational Autoencoder and decision ferns, enabling smooth traversal and applications in spatial transformations and kinematics prediction.

Contribution

It presents a new controller module for latent space traversal and an end-to-end trainable system that enhances manipulation of latent representations.

Findings

Effective latent space traversal demonstrated

Application to spatial transformations shown

Predictive kinematics in video sequences achieved

Abstract

The practice of transforming raw data to a feature space so that inference can be performed in that space has been popular for many years. Recently, rapid progress in deep neural networks has given both researchers and practitioners enhanced methods that increase the richness of feature representations, be it from images, text or speech. In this work we show how a constructed latent space can be explored in a controlled manner and argue that this complements well founded inference methods. For constructing the latent space a Variational Autoencoder is used. We present a novel controller module that allows for smooth traversal in the latent space and construct an end-to-end trainable framework. We explore the applicability of our method for performing spatial transformations as well as kinematics for predicting future latent vectors of a video sequence.

Figures

Tables

Table 1. Table 1: Mean squared error values on MNIST for rotation, dilation and rotation+dilation operations across CNN-baseline, DCIGN and LSTNet. Note that significant improvement LSTNet offers for the combined operation of rotation and dilation, indicating modularity.

Model	Rotation (MSE)	Dilation (MSE)	Rotation+Dilation (MSE)
DCIGN [12]	0.07373484	0.02599725	0.08349568
CNN-baseline	0.02819574	0.00841950	0.06508310
LSTNet	0.02380177	0.00835836	0.04410466

Table 2. Table 2: Steering angle prediction MSE on the KITTI test data.

Model	Steering Angle MSE (Degrees2)
Copy Last Frame	1.3723866
PredNet (FineTuned) [7]	2.4465750
LSTNet (Ours)	0.5267124

Equations

$$\begin{array}{r}\log P(X)-\mathcal{D}[\mathcal{Q}(z|X)||P(z|X)]=\\ E_{z\sim \mathcal{Q}}[logP(X|z)]-\mathcal{D}[\mathcal{Q}(z|X)||P(z)]\end{array}$$

$$q(\ell )=\frac{{\sum }_i\omega (\mathcal{D}(x_i))v_i}{n_{\ell }}$$

$$z_{t+h}=z_t+F(z_t,z_{t-1},z_{t-2},...,\theta )$$

$$\begin{array}{r}\frac{z_{t+h}-z_t}{h}=\frac{1}{h}F(z_t,z_{t-1},z_{t-2},...,\theta )\\ \frac{\partial z_t}{\partial h}=F(z_t,z_{t-1},z_{t-2},...,\theta )\end{array}$$

$$\begin{array}{r}{\mathbb{L}}_{VAE}=\frac{1}{N}\sum _{i\in \mathcal{B}}{\Vert \widehat{I_i}-I_i\Vert }^2+\frac{1}{N}\sum _{i\in \mathcal{B}}KL(z_i,\mathcal{N}(0,U)),\\ z\sim \mathcal{N}({\mu }_{enc}(I),{\sigma }_{enc}^2(I))\end{array}$$

$$\begin{array}{r}{\mathbb{L}}_z=\frac{1}{N}\sum _{i\in \mathcal{B}}{\Vert z_{target}-{\hat{z}}_{target}\Vert }^2,{\hat{z}}_{target}=C(z_{t-n},...,z_{t-1},\theta ),\\ z_{target}={\mu }_{enc}(I_t)\end{array}$$

$$\begin{array}{r}{\mathbb{L}}_I=\frac{1}{N}\sum _{i\in \mathcal{B}}{\Vert I_{target}-{\hat{I}}_{target}\Vert }^2,{\hat{I}}_{target}=\mathcal{P}(I_{t-n},...,I_{t-1},\theta ),\\ I_{target}=I_t\end{array}$$

$$d_n(\mathbf{x},\mathbf{t})=\sigma ((x_n-t_n))$$

$$\begin{array}{r}Q=q_0\times p_0\times p_1+q_1\times p_0\times (1-p_1)\\ +q_2\times (1-p_0)\times p_1+q_3\times (1-p_0)\times (1-p_1)\end{array}$$

$$Q=b+d_0\times x+d_1\times y+d_0\times d_1\times z$$

$$\begin{array}{r}{d}_0=\tanh (x_0),d_1=\tanh (x_1)\\ b=\frac{1}{2^h}\times (q_0+q_1+q_2+q_3),x=\frac{1}{2^h}\times (q_0-q_1+q_2-q_3)\\ y=\frac{1}{2^h}\times (q_0+q_1-q_2-q_3),z=\frac{1}{2^h}\times (q_0-q_1-q_2+q_3)\end{array}$$

$$C(z_{t-n},...,z_{t-1},\theta )=z_{t-1}+TN(z_{t-n},...,z_{t-1},\theta )$$

Full text

1 Introduction

A large part of human perception and understanding relies on using visual information from the surrounding environment and interpreting this information to subsequently act upon these interpretations. However, these interpretations are not applied directly to what is being seen; rather, there exists an abstraction mechanism from the image space into a more informative space so that complex inferences can be made there [1]. Similarly in machine learning, we would like machines to inherit this ability to abstract as it is the key to understanding and learning when it comes to real world data.

Recently, deep learning has demonstrated enormous success across various vision-related tasks such as image classification, object detection and semantic segmentation [2, 3, 4, 5, 6]. However, this level of success has yet to transition across to more complicated tasks such as video prediction. Many of the popular deep learning methods approach these more challenging tasks in a similar manner to image classification or segmentation, choosing to learn directly from the image space [7, 8]. This presents a challenge because often the image space is high-dimensional and complex; there exists a large semantic gap between the input pixel representation of the data and the desired transformation of said data for complex inference tasks such as prediction.

To address this challenge, we leverage the compact encoding space provided by a Variational Autoencoder (VAE) [9] to learn complex, higher order tasks in a more feasible way. Inference in the latent space takes advantage of solving a far more tractable problem than performing the operation in the image space as it gives a more natural way of separating the task of image construction and inferring semantic information. Other works have similarly utilised a compact encoding space to learn complex functions [10, 11, 12, 13]; however, even this encoding space can be strongly entangled and highly non-linear. As such, we construct a residual decision fern based architecture which serves as a controller module that provides the necessary non-linearity to properly disentangle encodings in the latent space. To this end, we introduce a novel framework for controlled traversal of the latent space called the Latent Space Traversal Network (LSTNet) and offer the following contributions:

• •

We discuss the benefits of operating in a latent space for complex inference tasks, introducing a novel decision fern based controller module which enables the use of control variables to traverse the latent space (Sections 4.1, 4.2 & 4.3).

• •

We create a unified, end-to-end trainable framework which incorporates our controller module with a residual VAE framework, offering an encoding space for learning high order inference tasks on real world data. Additionally, this framework offers a key insight into separating the tasks of pixel reconstruction and the high order inference (Sections 4.3 & 5.3).

• •

We demonstrate significant qualitative and quantitative improvements offered by LSTNet over popular models that impose geometrical and kinematic constraints on the prediction search space across the MNIST and KITTI datasets (Sections 5.1, 5.2 & 5.3).

2 Related Work

2.1 Learning Representations for Complex Inference Tasks

Operating in the latent space for demonstrating certain properties has been shown in several works. Applying a convolution architecture to a GAN framework [14] showed that following the construction of the latent space and applying arithmetic operations between two latent vectors observes the semantic logic we would expect from such an operation. InfoGAN [15] added a regularisation term which maximises mutual information between image space and some latent variables. Both these works claim to yield a disentangled latent vector representation of the high dimensional input data, demonstrating this by choosing a specific latent variable and interpolating across two values and showing smooth image transformation. However, due to their unsupervised nature, there are no guarantees on what attributes they will learn and how this will distort the intrinsic properties of the underlying data.

The work of  [12] divides the learned latent space in a VAE into extrinsic and intrinsic variables. The extrinsic variables are forced to represent controllable parameters of the image and the intrinsic parameters represent the appearance that is invariant to the extrinsic values. Although it is trained in a VAE setting, this method requires full supervision for preparing training batches. [11] introduces a fully supervised method, introducing a Gaussian Process Regression (GPR) in the constructed latent space of a VAE. However, using GPR imposes other limitations and assumptions that do not necessarily apply in training a VAE. [13] uses a semi-supervised approach for video prediction, training a VAEGAN [16] instead of a standard VAE. The VAEGAN leads to sharper looking images but imposes a discriminator loss which complicates the training procedure drastically. This work is similar to ours in that control variables are used to guide learning in the latent space. However, this method is not end-to-end trainable and uses a simple framework for processing the latent space; [7] showed this framework underperformed in next frame prediction and next frame steering prediction.

[8, 10] offered a video prediction framework which combines an adversarial loss along with an image gradient difference/optical flow loss function to perform next frame prediction and also multi-step frame prediction. Similarly, [7] performed predictive coding by implementing a convolutional LSTM network that is able to predict the next frame and also multi-step frame prediction in a video sequence. In contrast to our work and [13], these works optimise learning in the image space. This approach suffers from poor semantic inference (especially over large time intervals) and will be a focus of investigation in our work.

2.2 Decision Forests & Ferns

Decision forests are known for their flexibility and robustness when dealing with data-driven learning problems [17]. Random decision forests were first developed by [18], where it was found that randomly choosing the subspace of features for training had a regularising effect which overcame variance and stability issues in binary decision trees. Following this, various methods which added more randomness quickly followed which helped further stabilise training of decision trees [19, 20, 21]. Further work extended decision trees into a related method of decision ferns, finding use in applications such as keypoint detection [22, 23]. Recently, an emerging trend has seen the incorporation of decision forests within deep learning frameworks [24, 25, 26, 27], utilising the non-linear discriminating capabilities offered by deep decision trees.

3 Background

3.1 Variational Autoencoders

There are many works that approximate probability distributions. For a comprehensive review, refer to [28] and more recently [29]. [9] proposed a unified encoder-decoder framework for minimising the variational bound:

[TABLE]

It states that minimising the term $\mathcal{D}[\mathcal{Q}(z|X)||P(z)]$ by choosing $z\sim\mathcal{N}(\mu,\sigma)$ and imposing a normal distribution on the output of the encoder will result in approximating $P(X)$ up to an error that depends on the encoder-decoder reconstruction error. In practice, the encoder and decoder are constructed to have sufficient capacity to encode and decode an image from the given distribution properly, and also the choice of latent space to be in a high enough dimensionality to have the descriptive capacity to encode the image information needed for reconstruction.

3.2 Decision Forests & Ferns

Decision Trees

A decision tree is composed of a set of internal decision nodes and leaf nodes $\ell$. The decision nodes, $\mathcal{D}=\{d_{0},\cdots,d_{N-1}\}$, each contain a decision function $d(\bm{x};\theta)$, which perform a routing of a corresponding input based on its decision parameters $\theta$. The set of input samples $\bm{x}$ start at the root node of a decision tree and are mapped by the decision nodes to one of the terminating leaf nodes $\ell=\mathcal{D}(\bm{x},\Theta)$, which holds a real value $\bm{q}$ formed from the training data ($\Theta$ denotes the collected decision parameters of the decision tree). Hence, a real value $q$ in a leaf node will be denote by:

[TABLE]

where $n_{\ell}$ is the total count of training samples mapped to leaf node $\ell$, $\omega$ specifies the decision tree mapping of the sample to leaf $\ell$ and $v_{i}$ specifies the value of sample $i$.

Decision Ferns

A decision fern is related to a decision forest. Similarly to decision trees, decision ferns route samples from the root node to a leaf node $\ell$. The key difference is that all decisions are made simultaneously rather than in sequence. Thus, a decision fern contains a single root node which holds the parameters $\Theta$ of the entire decision fern to map from the root to a leaf node.

4 Operating in Latent Space

To operate in latent space, there must exist a mechanism which allows for transition between the latent space and image space. Hence, there is an inherent trade off between obtaining a good semantic representation of the image space via its corresponding latent space, and the reconstruction error introduced when transitioning back to the image space. In this work, we show that the benefits of working in a compact latent space far outweigh the loss introduced by image reconstruction. The latent space emphasises learning the underlying changes in semantics related to an inference task which is paramount when learning high order inference tasks.

4.1 Constructing a Latent Space

When constructing a latent space, several viable options exist as candidates. The most naive method would be to perform Principal Component Analysis (PCA) directly on images [30], selecting the $N$ most dominant components. Clearly, this method results in a large loss in information and is less than ideal. Generative Adversarial Networks (GANs) [14, 31, 32] can produce realistic looking images from a latent vector but lack an encoder for performing inference and are difficult to train. Techniques which combine GANs with Variational Autoencoders [16, 29, 33, 34, 35] offer an inference framework for encoding images but prove to be cumbersome to train due to many moving parts and instability accompanying their adversarial training schemes.

Hence, to construct our latent space, we use the relatively straightforward framework of a Variational Autoencoder (VAE) [9]. VAEs contain both a decoder as well as an encoder; the latter of which is used to infer a latent vector given an image. This encoding space offers a low dimensional representation of the input and has many appealing attributes: it is semi-smooth and encapsulates the intrinsic properties of image data. It is separable by construction; it maintains an encoding space that embeds similar objects in the image space near each other in the latent space. Furthermore, a VAE can be trained in a stable manner and in an unsupervised way, making it an ideal candidate to learn complex higher order inference tasks.

4.2 Traversing the Latent Space

Recent works attempted to learn the latent space transformation using various models, such as RNNs [13] and a Gaussian Process Regression [11]. In our work we recognise that although the original input data is reduced to a lower dimension encoding space, inference on this space is a complex operation over a space which if constructed correctly, has no redundant dimensions; hence all latent variables should be utilised for the inference task. Under the assumption of a smooth constructed manifold and a transformation that traverses this smooth manifold under a narrow constraint (in the form of a control variable or side information), a reasonable model for the controller module is:

[TABLE]

where $\{z_{t},z_{t-1},z_{t-2}...\}$ are the latent vectors corresponding to input data $\{x_{t},x_{t-1},x_{t-2}...\}$, $\theta$ is the control variable and $z_{t+h}$ is the output of the model corresponding to given the inputs. The operator $F(\mathbf{z},\theta)$ can be interpreted as:

[TABLE]

where $\frac{1}{h}$ can be absorbed into $F(\mathbf{z},\theta)$ and by doing so, we can interpret it as a residual term that is added for smoothly traversing from input $z_{t}$ to $z_{t+h}$, given side information $\theta$ and the history $\{z_{t},z_{t-1},z_{t-2}...\}$. This construction allows us to implement Eq. 4 using a neural network, which we denote as the Transformer Network. Eq. 3 encapsulates the complete controller, which is a residual framework that delivers the final transformed latent vector (denoted as the Controller Module). The Transformer Network, will be doing the heavy lifting of inferring the correct step to take for obtaining the desired result $\hat{z}_{t+h}$ and as such should be carefully modelled. In the following section, we discuss our chosen implementation and the considerations that were taken when constructing the Transformer Network.

4.3 Latent Space Traversal Network

Our Latent Space Traversal Network (LSTNet) consists of two main components:

A VAE with an encoder and decoder. The encoder learns a latent representation to encode the real set of training images into this space. The decoder learns a mapping from latent space back to the image space. 2. 2.

A Controller Module (C) with a Transformer Network (TN), that applies an operation in the latent space offered by the VAE.

An overview of our model is shown in Fig. 1. In the rest of this section, we detail and justify the choice of architecture for the components of our model.

To construct our latent space, we adopt the approach in [36] and construct a residual encoder and decoder to form our VAE. This residual VAE offers a low-dimensional, dense representation of input image data, allowing for a latent space which makes higher order inference tasks easier to learn in. LSTNet is trained on a loss function composed of three terms:

[TABLE]

[TABLE]

[TABLE]

where $~{}\mathbb{L}_{VAE}$ is the loss for the VAE which updates the encoder and decoder parameters, $~{}\mathbb{L}_{z}$ is the controller loss which updates the controller network’s parameters and $~{}\mathbb{L}_{I}$ is the predicted image loss which updates the controller and decoder of the VAE. In this case, $I$ is an image, $z$ is a latent vector in the encoding space, $\mathcal{B}$ is a minibatch, $U$ is an identity matrix, $\mu_{enc}$ and $\sigma_{enc}^{2}$ are the respective mean and variance of the encoder’s output, $C$ is the controller network and $\mathcal{P}$ denotes the LSTNet (passing an input image(s) through the encoder, controller and decoder to generate a transformation/prediction).

Fern-based Transformer Network

Even in the latent space, learning complex tasks such as video prediction can be difficult. In the experiments section, we motivate the use of the fern-based controller over a linear variant composed of stacked fully connected layers with ReLU non-linearities.

Our transformer network employs an ensemble of soft decision ferns as a core component. The use of soft decision ferns allows them to be differentiable such that they can be integrated and trained within an end-to-end framework. One way to achieve this is construct decision functions which apply a sigmoid to each input activation biased with a threshold value, yielding a soft value between $[0,1]$:

[TABLE]

where $\sigma(x)$ is a sigmoid function. $x_{n}$ and $t_{n}$ are the respective input activation and corresponding threshold values assigned towards the decision. To illustrate this, for a depth two fern using two activations which create the soft routes to its corresponding four leaves, its output $Q$ is:

[TABLE]

where $p_{0}$ and $p_{1}$ are the respective probability outputs of the decision functions of the decision ferns. $q_{0},q_{1},q_{2}$ and $q_{3}$ are the corresponding leaf nodes of the decision fern (illustrated in Fig. 2a). Eq. 9 can be reparameterised as:

[TABLE]

where:

[TABLE]

$x_{0}$ and $x_{1}$ are the assigned activations to the decision fern and $h$ is the fern depth. $b,x,y,z$ can be represented by fully connected linear layers which encapsulates all decision ferns in the ensemble.

Fig. 2b shows the architecture of our transformer network. We adopt the residual framework in [4], modifying it for a feedforward network (FNN) and adding decision fern blocks along the residual branch in the architecture. In Fig.2c, we outline the construction of a decision fern building block in the TN, consisting of ferns of two levels in depth. The decision nodes of the fern are reshaped from incoming activations, to which Batch Normalisation [37] and a Hyperbolic Tangent function is applied. This compresses the activations between the range of $[-1,1]$ and changes their role to that of making decisions on routing to the leaf nodes. A split and multiply creates the conditioned depth two decisions of the fern, which is concatenated with the depth one decisions. Finally, a FC linear layer serves to interpret the decisions made by the decision fern and form the leaf nodes which are free to take any range of values.

Hence, our final controller network is expressed as:

[TABLE]

5 Experiments

In our experiments, we explore operation sets that can be achieved by working in the latent space. We choose two applications to focus on - the first application looks towards imposing a spatial transformation constraint on the latent vector; the second application is a more complex one, looking at video prediction. For the first application, we use the MNIST dataset [38] as a toy example and investigate rotating and dilation operations on the dataset. For the second application, we use the KITTI dataset [39] and perform video prediction and steering prediction. For all experiments, we trained using the ADAM Optimiser [40] with learning rate of 0.0001, first and second moment values of 0.9 and 0.999 respectively, using a batch size of 64.

5.1 Imposing Spatial Transformation

For imposing spatial transformations, we present rotating and dilation operations and show how to constrain the direction a latent vector will traverse by using a spatial constraint. This constrained version of LSTNet applies a transformation to a single image which either rotates or erodes/dilates the given image. For both rotation and dilation experiments, we use a small residual VAE architecture along with our specified controller module with 1 residual layer with decision fern blocks (refer to Fig. 2b & 2c for details). We choose a latent vector size of 100 dimensions. The encoder consists of 2 residual downsampling layers (refer to [36] for details on the residual layers), Batch Normalisation and ReLU activations in between, ending with 2 fully connected linear layers for emitting the mean and variance of the latent vector. The decoder also consists of 2 residual upsampling layers, Batch Normalisation and ReLU activations in between, ending with a Hyperbolic Tangent function. To compare our method, we use two baselines. The first method is the most obvious comparison; we implement a baseline CNN which learns a target transformation, given an input image and corresponding control variable $\theta$ (CNN-baseline). This CNN-baseline is composed of 2 strided 3x3 convolution layers, 2 FC linear layers and 2 strided 3x3 deconvolution layers with ReLU non-linearities used for activation. The number of output channels in the hidden layers was kept at a constant 128. Additionally, we implement the Deep Convolutional Inverse Graphics Network [12] as specified in their paper for comparison. For each of the three methods compare under the same conditions by providing the control variable $\theta$ as an input during training and testing.

Rotation

We create augmented, rotated samples from the MNIST dataset. Specifically, we randomly choose 600 samples from the data, ensuring a even distribution of 60 samples per class label are chosen. For each sample, we generate 45 rotation augmentations by rotating the sample in the range of $-45^{\circ}<\theta<45^{\circ}$. We add this augmented set to the original MNIST data and train the VAE with controller module end-to-end for 20k iterations. To inject the control variable into the input, we concatenate it to the encoded latent vector before feeding it to the controller module.

To train our controller module, we note that there are $\binom{45}{2}$ possible pairs in each example giving us much more training data than needed. Hence, for every iteration of training, we randomly choose a batch of triplets $(I_{i},I_{j},\theta)$, where $\theta$ is a rotation control variable specifying the rotation in radians. For inference, we randomly sample images from the MNIST dataset that were not selected to be augmented and perform a rotation parameter sweep. This results in smooth rotation of the image, while preserving the shape (see Fig. 3a). Note that other works (i.e [15]) have shown that by altering a variable in the latent space, a rotated image can be retrieved, but fail to preserve image shape, leading to distortion. This indicates that the specific variable for rotation is not only responsible for rotation, but has influence over other attributes of the image. In contrast, we observed this difference across several variables between $z_{i}$ and $z_{j}$. This gives the insight that in order to perform a rotation (or similar spatial transform operation) and preserve the original image shape, several variables in the latent vector need to change which justifies the use of a highly non-linear network to approximate this operation.

Thickening

For learning the dilation operation we created augmentations in a similar manner to rotation operations. We randomly choose 5000 samples (evenly distributed across the 10 class labels) from the original dataset, and augmented every sample 4 times with 2 steps of dilation (thickening) and 2 steps of erosion (thinning). We train our VAE and controller module in a similar way to the rotation operation for 20k iterations, specifying batches of triplets $(I_{i},I_{j},\theta)$; $\theta$ changes its role to specifying a dilation factor that takes one of 5 discrete values in the range of $[-2,2]$. Note that although the network was trained on 5 discrete levels of dilation, it manages to learn to smoothly interpolate when performing the operation sweep during inference (as shown in Fig. 3a).

Combining Operations

One immediate extension that LSTNet offers when performing spatial transformation operations is the ease in which multiple spatial transformation operations can be combined together into a single framework. In Fig. 3b, we show the samples produced by an LSTNet with 2 controller modules, sharing a single latent space offered by the VAE. It is important to note here that neither the rotation or dilation controller modules saw the other’s training data. Hence, both operations are applied consecutively in the latent space and decoded back for visualisation.

In Table 1, we show the mean squared error (MSE) of LSTNet, comparing against the two baseline methods across rotation, dilation and combined rotation plus dilation operations. We can see that LSTNet outperforms in Rotation and Dilation MSE and handily outperforms in the combined operation. This indicates a generality in learning in the latent space; LSTNet has learned the semantics behind rotation and dilation operations and thus can seamlessly combine these two operations.

5.2 Imposing Kinematics

We now move towards the more complex inference task of video prediction. Similar to imposing spatial transformation, we use a larger, residual VAE architecture along with a larger controller module to account for the more complex inference task. We increase the base dimension of our latent vector to 256 dimensions. The VAE’s encoder consists of 3 residual downsampling layers, Batch Normalisation and ReLU activations in between, ending with a fully connected linear layer. The decoder consists of 3 residual upsampling layers, Batch Normalisation and ReLU activations in between, ending with a Hyperbolic Tangent function. For the controller network we test out two variants. The first is the fern-based controller as described in Section 4.3. For motivating the use of the fern-based controller, a linear variant controller network is also used: this is a feedforward linear network consisting of 4 FC linear layers with ReLU activations, matching the fern-based controller in terms of model capacity.

Similarly to imposing spatial transformations, we randomly create batches of triplets $(I_{i},I_{j},\theta)$, where $I_{i}$ and $I_{j}$ are the respective current frame and target future frame (randomly chosen to be within 5 time steps of the current frame) and $\theta$ is the corresponding time step from the current frame to the target frame. Our controller module receives the latent vectors of the current frame as well as a sequence of latent vectors belonging to the previous 5 frames to the current frame. We train our framework end-to-end for 150k iterations; the VAE is trained to minimise reconstruction loss along with KL regularisation, the controller is minimised using the latent error between the target latent and predicted latent vectors and both decoder and controller are jointly optimised using the error between target frames and predicted frames (refer to Eqs. 5, 6 and 7). The VAE is trained at a ratio of 5:1 against the controller module to ensure a proper representation of the input in its encoding space.

Fig. 6 shows qualitative results comparing our model with the PredNet model in [7]. We can see that the further the prediction is over time, the less accurate PredNet becomes, whilst LSTNet remains much more robust to changes in the scene over time. Observing the samples of PredNet, a recurring phenomena is that in areas of the frame where object movement should occur, moving objects are instead smeared and blurred. In the case of LSTNet, predicted movement is better observed (particularly over large time steps), whilst vehicles and street furniture (i.e road signs and markings) are better placed. This gives an indication that LSTNet is more reliable for prediction mechanism within the context of the task. PredNet implicitly learns to predict on a fixed timestep and hence predicts over a large time interval by rolling out over its predicted images. On the other hand, LSTNet offers a more general approach via the control variable $\theta$. It learns a transformation which allows it to directly predict the target video frame without the computational overhead of rolling out fixed timesteps to reach the target.

In Table 4a, we depict the average MSE and SSIM over 5 future frames (500ms time lapse) for the KITTI test set. We can see that LSTNet outperforms the compared methods of Copy Last Frame, PredNet and a Linear Controller Variant (as discussed in Section 4.3). Looking at Fig. 4b, we can see that LSTNet achieves lowest MSE for all timesteps, except $\hat{I}_{t+1}$ , where PredNet has slightly lower MSE. However, despite slightly higher initial MSE, the performance of LSTNet quickly exceeds both methods as inferring good prediction begins playing a larger factor over time (see Section 5.3 for a full discussion). These quantitative results correlate well with our qualitative results and again indicate that LSTNet is able to outperform on the task of prediction rather than on image reconstruction.

5.3 Latent Space for Prediction

In addition to the computational and memory footprint benefits, we show that projecting an image onto the latent space, operating on the latent vector using our controller module and reprojecting back into the pixel space has on average a lower MSE in the pixel space over operations that are increasingly harder to perform; for example: chaining spatial transformations and predicting more than 1 time step into the future.

Moving vs. non-moving objects

An attribute that does not favour a latent space framework is the inference of the fine grained detail in images with a lot of static, non-moving components. Fig. 5 shows an example of a $\hat{I}_{t+1}$ prediction for PredNet (Finetuned) [7], our LSTNet and the ground truth for comparison. Across these three images, red and green highlighted patches show texture and movements of objects respectively. Across the patches, it is visually apparent that LSTNet does not generate a fine detailed texture of the tree leaves, although it captures the movement of a car well as the camera viewpoint changes. Conversely, PredNet is able to capture the fine grain texture of the tree leaves, but fails to capture the movement of the stationary car from a camera viewpoint change.

We perform a simple patch-based test to illustrate the importance of emphasising prediction on moving objects versus non-moving objects. For $\hat{I}_{t+1}$, we randomly choose 20 patches of 20$\times$20 pixels that we identify where movement occurs; computing the average MSE yielded values for LSTNet=0.0020 and PredNet=0.0134 showing we significantly improve on predicting movement. Similarly, we sample 20 patches identified as being static with no moving objects which yielded an MSE for LSTNet=0.0024 and PredNet=0.0021, with the results favouring PredNet. This result correlates well with the competitive results shown on $\hat{I}_{t+1}$ in Fig. 4b; between two consecutive frames, the majority of parts in a scene are static with little to no movement. Hence, prediction plays a smaller role in pixel space MSE over such a short time frame. However, as the time between predicted and current frame increases, getting better predictions on movements plays a larger role, which accounts for the results shown in Fig. 4b.

Auxiliary Parameter Predictions

Furthermore, inferring auxiliary tasks such as steering angle does not require the fine detailed knowledge contained in the pixel space of a scene. For performing such inference tasks, the main requirement is the semantic information contained in the scene. For this task, we used a pretrained LSTNet and added a FC layer to the controller to output a single value and finetuned to learn the steering angle. This is where LSTNet shines; it considerably outperforms PredNet [7] and copying steering angles from the last seen frame as shown in Table 2. This further correlates with our MSE prediction results that LSTNet is inherently able to distill the semantics from a scene for complex inference tasks.

6 Conclusion

In this work, we present a novel, end-to-end trainable framework for operating on a latent space constructed using a VAE. We explore the power of learning in latent space on two operations: spatial transformations and video prediction, and show semantic gains for increasingly harder inference tasks which subsequently translates to a more meaningful result in the pixel space. Furthermore, as a direct extension to this work, the use of a VAE presents an opportunity to explore multi-model predictions for further robustness in predictive tasks.