Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold

TL;DR

This paper introduces a novel high-dimensional regression approach that combines Gaussian process regression with variational autoencoders to predict complex visual data responses on manifolds.

Contribution

It presents a new method for high-dimensional image response regression, learning latent space representations aligned with data space responses within a VAE framework.

Findings

Effective in high-dimensional visual data regression

Robust performance demonstrated on various datasets

Integrates Gaussian processes with VAEs for complex manifolds

Abstract

This paper proposes a new high dimensional regression method by merging Gaussian process regression into a variational autoencoder framework. In contrast to other regression methods, the proposed method focuses on the case where output responses are on a complex high dimensional manifold, such as images. Our contributions are summarized as follows: (i) A new regression method estimating high dimensional image responses, which is not handled by existing regression algorithms, is proposed. (ii) The proposed regression method introduces a strategy to learn the latent space as well as the encoder and decoder so that the result of the regressed response in the latent space coincide with the corresponding response in the data space. (iii) The proposed regression is embedded into a generative model, and the whole procedure is developed by the variational autoencoder framework. We demonstrate the robustness and effectiveness of our method through a number of experiments on various visual data regression problems.

Tables

Table 1. Table 1: Measure for the results with / without background.

Structural Similarity Index Measure [41] result
sports	Proposed	R-VAE [24]	MIGP [2]	NN
Baseball	0.610 / 0.607	0.492 / 0.489	0.803 / 0.247	0.215 / 0.210
Golf	0.752 / 0.707	0.578 / 0.543	0.845 / 0.114	0.244 / 0.213
Snatch	0.377 / 0.369	0.207 / 0.205	0.626 / 0.019	0.206 / 0.198

Table 2. Table 2: Measure for images from + 0.5 ​ σ , + 1.0 ​ σ 0.5 𝜎 1.0 𝜎 0.5\sigma,+1.0\sigma and + 1.5 ​ σ 1.5 𝜎 +1.5\sigma .

SSIM result for different standard deviations
sports	method	+0.5$\sigma$	+1.0$\sigma$	+1.5$\sigma$
Baseball	proposed	0.6453	0.5980	0.5307
	R-VAE [24]	0.4993	0.4402	0.3825
Golf	proposed	0.7203	0.4839	0.4422
	R-VAE	0.5642	0.4026	0.2417
Snatch	proposed	0.4042	0.3656	0.3629
	R-VAE	0.2700	0.1645	0.0770

Table 3. Table 3: Similarity measure for generated human pose image.

SSIM result for human pose generation
Actors	proposed(A)	CVAE(A)	proposed(B)	CVAE(B)
#1	0.7402	0.4849	0.5227	0.4059
#2	0.6743	0.4265	0.4775	0.3580
#3	0.7295	0.5094	0.5013	0.4268
#4	0.7671	0.4954	0.5224	0.4198

Equations

$$\begin{array}{rl}L(\theta ,\varphi )=& -D_{KL}(q_{\varphi }(z|x,y)||p_{\theta }(z))\\ +& \sum _{i=1}^N\log p_{\theta }(y_i|z_i)+\sum _{j=1}^M\log p_{\theta }({\hat{y}}_{\ast j}|z_{\ast j}).\end{array}\text{vspace}-3mm$$

$$q_{\varphi }(z|x,y)=\mathcal{N}(z|m(x,y),\sigma (x,y)).$$

$$m_{\ast ,j}=K_{\ast ,j}{K}^{-1}\text{Z},{\sigma }_G=(K_{\ast \ast ,j}-K_{\ast ,j}{K}^{-1}{K}_{\ast ,j}^T)I.$$

$$K=\left[\begin{array}{ccc}k(x_1,x_1)& \cdots & k(x_1,x_N)\\ ⋮& \ddots & ⋮\\ k(x_N,x_1)& \cdots & k(x_N,x_N)\end{array}\right],$$

$$K_{\ast \ast ,j}=k(x_{\ast ,j},x_{\ast ,j}),$$

$$K_{\ast ,j}=[k(x_{\ast ,j},x_1),k(x_{\ast ,j},x_2),\cdots ,k(x_{\ast ,j},x_N)].$$

$$q_{\varphi }(z|x,y)=\{\begin{array}{ll}\mathcal{N}(m_i,{\sigma }_i)& x=x_i,y=y_i.\\ \mathcal{N}(m_{\ast ,j},{\sigma }_{\ast ,j})& x=x_{\ast ,j},y\in \varnothing .\end{array}$$

Full text

1 Introduction

Regression of paired input and output data with an unknown relationship is one of the most crucial challenges in data analysis. In diverse research fields, such as trajectory analysis, robotics, the stock market, etc. [15, 44, 44, 28, 22], target phenomena are interpreted as a form of paired input / output data. In these applications, a regression algorithm is usually used to estimate the unknown response for a given input by using the information obtained from the observed data pairs. Many vision applications can also be expressed as such input / output data pairs. For example, in Fig. 1 (a), the sequence of the motion images can be described by input / output paired data, where the input can be defined as the relative order and the output response is defined as the corresponding image. The motion capture data and their corresponding images in Fig. 1 (b) are another example. The input data are 3D joint positions, and their responses will be the corresponding posture images. If we can model the implicit function representing the given image data pairs via regression, we can estimate unobserved images that correspond to the input data.

However, applying existing multiple output regression algorithms [1, 3, 2, 39] to these kinds of visual data applications is not straightforward, because the visual data are usually represented in high dimensional spaces. In general, high dimensional visual data (such as image sequences) are difficult to be analyzed with classical probabilistic methods because of their limited modeling capacity [17, 18]. Thus, regression of visual data to estimate visual responses requires a novel approach.

In handling high dimensional complex data, recent attempts [12, 27, 37, 24, 36] using a deep generative network, such as the variational autoencoder (VAE) [24], have achieved significant success in reconstructing images. In VAE, a latent variable is defined to embed the compressed information of the data, and an encoder is trained to map a data space into its corresponding latent space. A decoder is also trained to reconstruct images from a sampled point in the latent space. The projection for the input data into the latent space (via the encoder) captures the essential characteristics of the data and allows execution of the regression task using a much lower dimensional space. The regression in the latent space is done together with the training of the encoder and decoder. However, a naive combination of regression and the VAE is not particularly effective because the decoder and the latent space are not designed in a way that permits the result of the regressed response in latent space, and the corresponding response in data space, to coincide. Therefore, a new method to simultaneously train the latent space and the encoder/decoder is required to achieve coincidence between the regressed latent vector and the reconstructed image.

In this paper, we solve this problem by combining the VAE [24] and Gaussian process regression [35]. The key idea of this work is to do regression in the latent space instead of the high-dimensional image space. The proposed algorithm generates the latent space that compresses the information of both the domain and output image using the VAE, and projects the data pairs to the latent space. Then, regression is conducted for the projected data pairs in latent space, and the decoder is trained so that the regression in latent space and image space coincide. To the best of our knowledge, it is the first attempt to apply the VAE framework to solve the regression problem. The whole process, including the loss function, is designed as the generative model, and a new mini-batch composition method is presented for training the decoder to satisfy our purpose.

All connection parameters of the encoder / decoder are inferred by the end-to-end stochastic gradient descent method as described in [23]. The proposed regression method is validated with two different examples: sports sequences and motion image sequences with skeletons. The first example presents a regression case of simple domain to complex codomain, and the second example presents the complex domain to complex codomain case.

2 Related Work

**Deep Generative Network: ** Classical probabilistic generative models [34, 8, 42, 19, 29, 40, 5] have proven to be successful in understanding diverse unsupervised data, but their descriptive ability is insufficient to fully explain complex data such as images [18]. Recently, as in other works in the vision area [16, 31], deep layered architectures have been successfully applied to solve this problem with powerful data generation performance. Among these architectures, generative adversarial network (GAN) [12] and generative moment matching networks (GMMN) [27] directly learn the generator that maps latent space to data space. Meanwhile, the variants of the restricted Boltzman machine (RBM)  [17, 18, 38, 37] and probabilistic autoencoders [24, 30, 11] learn the encoder that defines the map from data to latent space and the generator (decoder) simultaneously. The former methods, and especially variants of GAN [12, 9, 33], are reported to describe the edges of generated images more sharply than the latter methods. However, the applicability of these methods is restricted due to the difficulty of discovering the relationships between data and latent space. This innate nature makes it difficult to use adversarial networks for designing the regression. Therefore, this paper adopts the variational autoencoder framework [24], which is also more suitable than RBM families to expand the regression model.

**Variational Autoencoder: ** Since Kingma et al. [24] first published the variational autoencoder (VAE), numerous applications [32, 43] have been presented to solve various problems. Yan et al. [43] proposed conditional VAE in order to generate the image conditioned on the attribute given in the form of sentences. Furthermore, recent work [14, 25, 13] has demonstrated that a sequence in latent space would be mapped back to the sequence of data. Hence these methods embedded dynamic models such as recurrent neural networks [14] and the Kalman filter [25, 21] into the VAE framework. These algorithms [14, 25, 25, 13] successfully show the ability of dynamic models in a latent space to capture the temporal changes of relatively simple objects in images. In this paper, we apply the VAE for the regression task in a relatively complex manifold.

**Regression: ** Regression of paired data is theoretically well established and analytic solutions for the infinite dimension of the basis function [4, 35] have been derived for the last century. In non-parametric cases, Gaussian process [35, 26] provides a general solution by expanding Bayesian linear regression using kernel metrics and a Gaussian process prior. By using this method, we can estimate the output data as a Gaussian posterior composed of given data pairs and input data to the unobserved target outputs. However, applying the algorithms for the high-dimensional output data is difficult because the kernel metric has limited capacity to express the complicated high dimensional data. The variants of multiple output regression algorithms [2, 3, 1, 39] are proposed to deal with multi-dimensional output responses. Still, these algorithms focus on handling relatively low dimensional output responses and are not able to sufficiently describe complicated data, such as that of an image. In this paper, we construct a regressed latent space by using a variational autoencoder to handle complex data.

3 Proposed Method

3.1 Overall Scheme

Given the target data pair $(x_{i},y_{i}),i=1,\ldots,N,x_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}$, our goal is to find the unknown response $y_{*}$ for a new input $x_{*}$. In this paper, the response $y\in\mathcal{Y}$ is defined as an image and the corresponding input $x\in\mathcal{X}$ is defined accordingly based on the applications as in Fig. 1.

As shown in Fig. 2, for the observed data pairs $(x_{i},y_{i}),i=1,\cdots,N$, the encoder/decoder produces $\hat{y_{i}}$ which is the reconstruction of an observed image $y_{i}$. For the observed data, the encoding network $E(\cdot)$ produces mean and variance for a part of the latent vector $z_{i}$, that is, $[m_{i,y},\sigma_{i,y}]=E(y_{i};W_{E})$ which compresses $y_{i}$ to a latent variable with Gaussian mean $m_{i,y}$ and variance $\sigma_{i,y}$. The remaining part of $z_{i}$ is modeled by $[m_{i,x},\sigma_{i,x}]=f(x_{i},W_{x})$ which represents mean and variance of $z_{i}$. Thus, the Gaussian distribution of $z_{i}$ is described by $m_{i}=[m_{i,y},m_{i,x}]$ and $\sigma_{i}=diag[\sigma_{i,y},\sigma_{i,x}]$. For the unobserved image $y_{*}$ for a newly given $x_{*}$, the proposed method produces $\hat{y}_{*}$, which is an estimator of $y_{*}$.

Using $z_{i}$ sampled from $\mathcal{N}(m_{i},\sigma_{i})$, the decoding network reconstructs the output response $\hat{y}_{i}$, that is, $\hat{y_{i}}=D(z_{i};W_{D})$. Note that if $(W_{E},W_{x},W_{D})$ is well trained by the training scheme in Section 3.4, $\hat{y_{i}}$ should be similar to $y_{i}$. However, for an unobserved $y_{*}$ to a given $x_{*}$, it is impossible to obtain $z_{*}$ from $E(\cdot;W_{E})$ because we do not have any information on $y_{*}$. To estimate $y_{*}$, we obtain $z_{*}$ by using regression from $\textbf{x}=\{x_{1},x_{2},\cdots,x_{N}\},\textbf{z}=\{z_{1},z_{2},\cdots,z_{N}\}$ and $x_{*}$. For this regression, $z_{i}$ is sampled from $\mathcal{N}(m_{i},\sigma_{i})$ for each observed response $y_{i}\in\textbf{y}=\{y_{1},y_{2},\cdots,y_{N}\}$. Then, we estimate $z_{*}$ using Gaussian process (GP) regression $z_{*}\sim R(x_{*};\textbf{x},\textbf{z},\sigma_{k})$ to be described in Section 3.2, where $\sigma_{k}$ is a kernel parameter of the GP regression, which can be produced by an additional encoder $\sigma_{k}=E^{\prime}(\textbf{y},W_{G})$; with $\sigma_{k}=[\sigma_{k,1},\cdots,\sigma_{k,N}]$. In this paper, for computational simplicity, we combine this kernel encoder with the encoder network $E(y;W_{E})$ and change the outputs into $[m_{i,y},\sigma_{i,y},\sigma_{i,k}]=E(y_{i},W_{E})$.

After $z_{*}$ is estimated, the response $\hat{y}_{*}$ is reconstructed from $z_{*}$ by using the decoding network $D(z_{*};W_{D})$. Note that the $D(z_{*};W_{D})$ should reconstruct not only $\hat{y}$ from the $z_{i}$ sampled by $\mathcal{N}(m_{i},\sigma_{i})$, but also $y_{*}$ from the $z_{*}$ which is the regression result obtained from $x_{*},\textbf{x},$ and y. The whole procedure is designed as a generative framework with joint distribution $p(x_{*},y_{*},\textbf{x},\textbf{y},W_{E},W_{x},W_{D})$, and hence can be derived by the VAE algorithm.

3.2 Variational Autoencoded Regression

The proposed scheme (depicted in Fig. 2) is derived from the directed graph model in Fig. 3. The diagram in Fig. 3 (a) represents the generative model describing a typical reconstruction problem, and the diagram in Fig. 3 (b) is the variational model which not only approximates the generative model in Fig. 3 (a), but also performs the regression for the estimation of unobserved $y$ by utilizing an information variable $x$ related to $y$. The joint distribution $p_{\theta}(y,z)$ can be expressed by the likelihood function $p_{\theta}(y|z)$ and the prior distribution $p_{\theta}(z)$, where $\theta$ refers to the set of all parameters related to the generation of the response $y\in\mathcal{Y}$ from the latent variable $z$. In our method, the prior distribution of $z$ is defined as zero mean Gaussian distribution, as in typical variants of VAE [43, 24]. Also, the likelihood function $p_{\theta}(y|z)$ depicts the decoding process in the proposed scheme. Below, it is shown that $\theta$ is realized by the parameter $W_{D}$ of the decoding network.

Once the joint distribution $p_{\theta}(y,z)$ is defined, the posterior $p_{\theta}(z|y)$ can be theoretically derived from the Bayes theorem, but the calculation is intractable. Therefore, the variational distribution $q_{\phi}(z|x,y)$ is introduced to approximate the true posterior distribution $p_{\theta}(z|y)$. Unlike $p_{\theta}(z|y)$, $x$ is introduced for the variational distribution $q_{\phi}(z|x,y)$ to sample $z_{*}\sim R(x_{*};\textbf{x},\textbf{z},\sigma_{k})$, which is the result of the GP regression for the unknown $y_{*}$. $q_{\phi}(z|x,y)$ represents the overall encoding procedure generating the latent variable $z$ from the input data pair $(x,y)$ and correspondingly, the variational parameter $\phi$ is realized by the parameters $W_{E},W_{x}$ as described in Section 3.1. Importantly, $q_{\phi}(z|x,y)$ should be able to explain both cases: 1) an observed image $y_{i}$, and 2) an unobserved image $y_{*}$ which requires regression as mentioned previously. For the first case, the variational distribution is defined as $z_{i}\sim q_{\phi}(z|x=x_{i},y=y_{i})$. For the latter case, the variational distribution is defined as $z_{*}\sim q_{\phi}(z|x=x_{*},y\in\varnothing)$ which represents the GP regression procedure for estimating latent $z_{*}$ for the input $x_{*}$.

In order to estimate the parameters $\theta$ and $\phi$ which minimize the distance between $p_{\theta}(z|y)$ and $q_{\phi}(z|x,y)$, we minimize the Kullback$-$Leibler divergence $D_{KL}(p_{\theta}(z|y)||q_{\phi}(z|x,y))$. Following the derivation in [6, 24], the minimization procedure $\{\theta^{*},\phi^{*}\}=\operatorname*{arg\,min}_{\{\theta,\phi\}}D_{KL}(p_{\theta}(z|y)||q_{\phi}(z|x,y))$ is converted to $\{\theta^{*},\phi^{*}\}=\operatorname*{arg\,min}_{\{\theta,\phi\}}L(\theta,\phi)$, where

[TABLE]

$z_{*,j}$ and $y_{*,j}$ represents the $M$ number of latent codes and output responses for $x_{*,j},j=1,\cdots,M$, to be regressed. In (1), $\hat{y}_{*,j}=D(z_{*,j};W_{D})$ is the reconstructed response from $z_{*,j}$ by the decoding network as depicted in Fig. 2. The parameters $\theta$ and $\phi$ are realized by the connection parameters of the encoding network with regression for $q_{\phi}(z|x,y)$, and the decoding network for $p_{\theta}(z|y)$ (see Section 3.3). To minimize the loss in (1), we propose a method for mini-batch learning (see Section 3.4). The Adam optimizer [23] is used for stochastic gradient descent training.

3.3 Model Description

For the encoding part, we define $q_{\phi}(z|x,y)$ which maps the data pair $(x,y)$ into the latent space $\mathcal{Z}$. For both, observed and unobserved images, $q_{\phi}(z|x,y)$ is defined by Gaussian distribution as in (2) and it enables us to analytically solve the KL-divergence term $D_{KL}(q_{\phi}(z|x,y)||p_{\theta}(z))$ in (1) following [24]:

[TABLE]

The variational parameter $\phi$ consists of the Gaussian mean function $m(x,y)$ and the variance function $\sigma(x,y)$. The $m(x,y)$ and $\sigma(x,y)$ are produced in different ways depending on the input data. When the input data is given by $x=x_{i}\in\textbf{x}$, the encoder yields $m(x,y)=[m_{i,y},m_{i,x}]$ and $\sigma(x,y)=diag[\sigma_{i,y},\sigma_{i,x}]$, where $diag[\cdot]$ refers to a diagonal matrix. When the input data is given by $x=x_{*,j}\in\textbf{x}_{*}$, $m(x,y)$ and $\sigma(x,y)$ are determined by the mean and variance $(m_{*,j},\sigma_{*,j})$ estimated by GP regression from z, x and $x_{*,j}$, where

[TABLE]

Z refers to the matrix $[z_{1};z_{2};\cdots;z_{N}]\in\mathcal{R}^{N\times D}$, and $I\in\mathcal{R}^{D\times D}$ is the identity matrix, where $D$ is the dimension of $z\in\mathcal{Z}$. The matrices $K,K_{**,j}$ and $K_{*,j}$ are defined as

[TABLE]

For the kernel $k(\cdot,\cdot)$, we use a simplified version of SE-kernel [35], where $k(x_{i},x_{j})=\sqrt{\sigma_{i}\sigma_{\textbf{}j}}\exp{{||x_{i}-x_{j}||}^{2}}$. Eventually, the variational parameter $\phi$ is realized by the weight matrices $(W_{x},W_{E})$ of the encoder network. In summary, for the given data $\textbf{x},\textbf{y},$ and $\textbf{x}_{*}$, $q_{\phi}(z|x,y)$ in (2) is given as

[TABLE]

For the decoding procedure, we define the likelihood function $p_{\theta}(y|z)=p(y|D(z;W_{D}))$, where $p(y|D(z;W_{D}))$ is defined as a Gaussian distribution with mean $D(z;W_{D})$ and fixed variance. Since the prior of $z$ is defined with zero mean Gaussian and identity covariance matrix, the weight $W_{D}$ represents the generative model parameter $\theta$. Correspondingly, the meanings of the second term and third term in (1) are interpreted as follows. Since the negative log-likelihood $(-\log(p_{\theta}(y|z)))$ is defined as $l_{2}$ distance ${||y-D(z;W_{D})||}^{2}$ in our algorithm, the second term represents the reconstruction error for the given data pair $(x_{i},y_{i})$ to $y_{i}$, and the third term denotes the estimation error for $y_{*,j}$ via regression from the given input data $x_{*,j}$ and the observed data $(x_{i},y_{i}),i=1,\ldots,N$.

3.4 Training

To train the parameters of the proposed model, a sufficient number of the training datasets is required. In our algorithm, a total of $V$ different training sequences $(x^{v}_{i},y^{v}_{i}),v=1,\ldots,V,i=1\ldots,N_{v}$ are used, as shown in Fig. 4. These training data pairs share similar semantics to the target (test) data pair $(x_{i},y_{i})$. If the target data pair is a golf swing sequence, the training data pairs will be different golf swing sequences obtained in different situations. Once the parameters are trained by the training dataset consisting of diverse golf swings, the proposed method can complete the target image sequence via regression from the incomplete test sequence on a golf swing. After training the model with the mini-batch, we fine-tune the parameters with observed data pairs in target regression.

**Mini-Batch Training: ** The work in [7] reports that the composition of the mini-batch is critical when using variants of stochastic gradient descent methods [23, 10] to train the parameters. To generate the batch, in this paper, $K$ sequences of a total $V$ sequences are randomly selected. For each selected training sequence $k=1\cdots K$, we randomly pick $L$ data pairs $(x^{k}_{l},y^{k}_{l}),l=1,...,L$, where $L=(M+N)$. For the earlier $N$ data pairs $(x^{k}_{n},y^{k}_{n}),n=1,\cdots,N$, we get the latent space vector $z^{k}_{n}$ from the encoder function $E(y^{k}_{n};W_{E})$, and $f(x^{k}_{n};W_{x})$ to train $W_{E},W_{x}$, and $W_{D}$. Alternatively, for the latter $M$ data pairs $(x^{k}_{m},y^{k}_{m}),m=(N+1),\cdots,L,$ we obtain the latent $z^{k}_{m}$ by regression (Section 3.3) from $\{z^{k}_{1},\cdots,z^{k}_{N}\}$, $\{x^{k}_{1},\cdots,x^{k}_{N}\}$ and $x^{k}_{m}$. The responses $\{y^{k}_{N+1},\cdots,y^{k}_{L}\}$ are assumed to be unknown in the encoding process. This data set is used to train the decoder network $D(z;W_{D})$ to reconstruct the proper responses not only for the $z^{k}_{n}$ from the data pair $(x^{k}_{n},y^{k}_{n})$, but also for the $z^{k}_{m}$ which are obtained from the regression. The corresponding loss from the estimated $\hat{y}^{k}_{m}$ and the actual $y^{k}_{m}$ refers to the the third term in (1). We note that it is possible to calculate the loss term because $y_{m}^{k}$ can be used as ground truth regression response. After constructing the batch, the stochastic gradient [23] for the batch is calculated to train all parameters.

**Parameter Fine-Tuning: ** After training the parameters $W_{E},W_{x}$ and $W_{D}$ using the batch from the training dataset, we further fine-tune the parameters with the observed data pairs $(x_{i},y_{i}),i=1,\cdots,N$ in the same way as previous regression techniques [35, 26]. Note that the training of the regression part is not done because the ground truth is not available for the test dataset. For the fine-tuning process, mini-batches are composed of the observed test data pairs $({x}_{i},{y}_{i})$ and randomly selected $(K-1)$ data sequences $(x^{k}_{i},y^{k}_{i})$ from the training set as in Fig. 5, where $i=1,\cdots,N_{k}$ and $k=1,...,(K-1)$. When the total number $N$ of observed test data pairs is less than $L$, we increase the number of samples by allowing repetition. Then, the parameters are fine-tuned with $50$ iterations. The detailed implementation is described in the supplementary material.

4 Experiments

In the experiments, we evaluated the regression capability of the proposed method via two applications composed of image data: (1) a problem with a simple temporal domain and complicated codomain and (2) a problem with a complicated domain and codomain. For the first application, we used sport data sequences obtained from YouTube. Human pose reconstruction for a given skeleton was tested for the second application.

4.1 Sports Data Sequences

Evaluation Scenario: In this scenario, we created data sets for three sport sequences: baseball swing, golf swing, and weightlifting. The dataset includes $236$ baseball swings, $232$ golf swings and $129$ weightlifting sequences from YouTube. In the dataset, $1000-2000$ images are included for each action sequence, and their relative orders are given. The domain is defined as $\mathcal{X}:[0,1]$ and a point in $\mathcal{X}$ is assigned to $x$ for each image $y\in\mathcal{Y}$ according to its relative order in the entire sequence. For testing, the golf and the baseball swings were trained with $200$ randomly selected sequences and tested with those that remained. The weight lifting scenario was trained with $100$ sequences. We executed the regression for each test sequence with $20$ observed images within all images of each sequence and compared the results with multiple-output GP regression (MOGP) [2], and GP regression combined with vanilla VAE [24] (called R-VAE from here on). For R-VAE, we conducted the fine-tuning process in the same way as the proposed method. For MOGP, we trained the kernel with two-thirds of the images in the given sequences.

Qualitative Analysis: Fig. 6 shows the qualitative comparison of image generation results. The sequences in Fig. 6 show samples uniformly picked among the regressed responses from $100$ evenly divided points in the range $[0,1]$. As seen in (a), the proposed method generated the most accurate responses compared to the other methods. R-VAE also succeeded in capturing the blunt characteristics of the background and the motions of the actions. However, the generated images in (b) suffer from large amount of noise for some images it is difficult to recognize the motion (circled in red). Demonstrated are also instances in which the order of the image was not matched (circled in blue), and instances in which the background of the image was not matched (circled in green). The images in the box show the samples of reconstruction results for given image pairs. Both, the proposed method and R-VAE successfully reconstructed the images, but the regression performance is largely different. As in (c), MOGP was not successful in describing the motion changes in the image, where every regression converged to the average of the training images.

We also conducted the experiments comparing with Nearest Neighbor (NN) method results to the proposed method. We investigated the reconstruction results by applying the NN to the latent space after VAE learning. However, the latent space encoded by the vanilla VAE is not appropriate enough to perform regression using the NN (see Fig. 7). This is because the encoding of the background region plays a dominant role in NN as compared to the motion region. This problem is clearly seen at the bottom right sequence in the Fig. 7. Although the background (green and sky) region is relatively similar to the observation, the swinging human regions are not correctly regressed. This clearly shows that the proposed regression in the latent space is well performed achieving expected regression results in the image space. The encoder and decoder are trained to link the regression results in the latent space to the regression results in the image space directly, which is not trivial as shown in Fig. 7 (b). This is the second contribution of the proposed method (Abstract-ii).

Fig. 8 shows the effect of the fine-tuning process. The first and second column show the results with and without fine tuning. The result of the proposed method is shown in the first row and that of R-VAE in the second row. Before fine tuning, both methods generated noisy outputs, but the proposed method captured the vast characteristics of the background as well as the change of the motions. In R-VAE, background information was less accurate than the proposed method (circled in red). After the fine-tuning process, both methods accurately reconstructed the given image pairs, as in (c). Nevertheless, the regression performance between the methods varied significantly, as in (b).

Fig. 9 represents the image generation results for different standard deviations. As with the original GP regression, the proposed method estimates the output responses in the form of mean and variance because the latent $z$ for reconstructing the image is sampled from Gaussian distribution, as in (7). As seen in (a), the proposed algorithm captured the core semantics of the motion in each image despite the deviation change. In R-VAE, the regression results were plausible when the sampled latent $z$ was close to the mean, but the motion in the image was regressed by a totally different action when adding large amounts of noise (up to 1.0$\sigma$). From this result, we can see that R-VAE also has an ability to align the images in the latent space according to their order as reported in previous works [24, 13]. However, the results also show that the learned variance of R-VAE does not represent the motion semantics required for regression well, which is essential for the realization of GP regression in the image space.

Quantitative Analysis: The quantitative performance was measured using the Structural Similarity Index Measure (SSIM) [41] which captures the structural similarities between two images. We estimated the $100$ images in the test set by using their domain information only, and compared the similarity between the ground truth image and the regression results. Table 1 shows the performance measures for generated regression images. For the three different sport sequences, the proposed method generated more similar images to the ground truth (GT) compared to R-VAE. Interestingly, the results of MOGP [2] which converged to the average of the images were measured to be most similar among the tested methods when including the background. This is because the background of the average image is almost the same as the background of the GT when the background of the GT is fixed. When we measured the similarity without the background region, MOGP was not successful and the proposed algorithm achieved the highest performance. Also, as with the result shown in Fig. 7, NN method marked unsuccessful quantitative performance. Table 2 and Fig. 9 show the performance when changing the standard deviation. We confirmed that the proposed method generated more plausible output than R-VAE for all cases.

4.2 Human Pose Reconstruction

Evaluation Scenario: For this experiment, we have used the human 3.6 million (H3.6m) [20] dataset for generating proper human appearance given the joint positions. The dataset provides $32$ joint positions, and thus the input data lie in $96$ dimensional space. The dataset includes diverse actions, and each action is repeatedly performed by different actors. Our goal is to estimate the proper image of a new skeleton by utilizing the observed pairs of joint positions and images. In the experiment, we used the ‘greeting’ and ‘posing’ scenarios of the H3.6m dataset. The scenario for each actor was captured in $8$ different view-points, resulting in a total of $16$ human pose sequences available for each actor. We trained the model with the motions of $4$ different actors using $12$ sequences from each actor. Then, we picked the observations from the remaining four sequences and conducted the regression. The joint vectors for the regression were selected from the sequences from which the observations were selected. The joint vectors from other actors were also tested. For comparison, we used the recent conditional VAE (C-VAE) [43] method, which generated an image according to a given attribute coupled with the sampled latent code. In this experiment, the joint vector was used as the attribute.

Qualitative Analysis: Fig. 10 (A) shows the pose generation result of the proposed algorithm and C-VAE. For C-VAE $(1)$, we used randomly sample latent code $z_{y}$ as in [43]. For C-VAE $(2)$, the latent code was given by the proposed regression block in Fig. 2. As shown in (c), the image regressed by the proposed method successfully describes the overall motion of each human pose. Also, note that the background of each image was correctly generated according to the view point of the observed data pair. The generated images from C-VAE $(2)$ contain a large amount of noise, but they captured the rough silhouette of the actors. This result is noticeable because C-VAE usually deals with cases in which the attribute is discrete. The result from C-VAE $(2)$ was clearer than the result from C-VAE $(1)$, but the difference was not significant. The result in Fig. 10 (B) shows the output responses when the joint vectors of other actors were given. The images in the blue box refer to the ground truth pose, and the images in the red box are the regression result by the proposed method. This result shows that the proposed method generates poses that resemble those of the input joint vectors while preserving the appearance of the given data pairs via regression. Specifically, when the given pair involves a man wearing white clothes, the generated image illustrates a man wearing the same clothes with a similar pose to the GT image. C-VAE (2) was not successful in generating a corresponding pose for a given joint from other actors.

Quantitative Analysis: Table 3 shows the similarities between the generated image and the ground truth image. The first two columns denoted by (A) represent the quantitative results in experiment (A) of Fig. 10. In the experiment, the proposed method achieved a higher score than C-VAE (2). For experiment (B), we compared the similarity between the regressed image and the original images for the joint vector (green box in Fig. 10). There, our method also achieved a higher score than C-VAE (2).

In this experiment, the input data lay in the high dimensional space and the target joint vectors were selected without considering temporal information. Despite the complicated and non-sequential input domain, the proposed regression method achieved reasonable output responses describing the semantics given in the input and the identity information contained in the observed pairs. It means that the proposed method is available for the temporal input and can also handle more complex and non-sequential input.

5 Conclusion

In this paper, we have proposed a novel regression method regarding high dimensional visual output. To tackle the challenge, the proposed regression method is designed so that the result of the regressed response in a latent space and the corresponding response in the data space coincide. Through qualitative and quantitative analysis, it has been verified that our method properly estimates the regressed image responses and offers an approximation of the complicated input-output relationship. This paper discovers meaningful progress in the regression field in that our work introduces a way to combine a deep layered architecture to the regression method in a probabilistic framework.

6 Acknowledgment

This work was partly supported by the ICT R&D program of MSIP/IITP (No.B0101-15-0552, Development of Predictive Visual Intelligence Technology), the SNU-Samsung Smart Campus Research Center at Seoul National University, EU FP7 project WYSIWYD under Grant 612139 and the BK 21 Plus Project. We thank the NVIDIA Corporation for their GPU donation.

Appendix A Implementation Detail

In the experiments, the encoder $[m_{i,y},\sigma_{i,y}]=E(y_{i};W_{E})$, decoder $D(z;W_{D})$ and $\sigma(y;W_{E})$ of the kernel for GP regression in Figure 2 of the paper are defined as multi-layered perceptrons. The encoder $E(y;W_{E})$ is designed with five convolution layers and one fully connected layer (the convolution layers are composed of $16,32,64,128,256$ channels with filter size $5\times 5$ each). All $y\in\mathcal{Y}$ are resized to three channel $64$-by-$64$ images. We set the dimension of the $m_{i,y}$ and $\sigma_{i,y}$ to $128$, and the fully connected layer returns the $256$ elements for $m_{i,y}$ and $\sigma_{i,y}$. The former $128$ elements are used as $m_{i,y}$, and the latter $128$ entries are defined as $\sigma_{i,y}$. For the mapping function $[m_{i,x},\sigma_{i,x}]=f(x_{i},W_{x})$, $m_{i,x}$ refers to $x_{i}\in\mathcal{R}^{n(\mathcal{X})}$ and the additional $n(\mathcal{X})$ outputs in $E(y;W_{E})$ indicates $\sigma_{i,x}$, as in Figure 2 of the paper. Therefore, the overall dimension of the final fully connected layer is $256+n(\mathcal{X})+1$; $256$ dimensions for $[m_{i,y},\sigma_{i,y}]$, $n(\mathcal{X})$ dimensions for $\sigma_{i,x}$, and one dimension for $\sigma_{k}$. For the decoding function $\hat{y}=D(z;W_{D})$, $6$ convolution layers with $2$-by-$2$ upsampling are used to reconstruct the image. The convolution layers have 256, 128, 64, 32, 16, 3 channels with filter size $4\times 4,5\times 5,5\times 5,5\times 5,5\times 5$ and $5\times 5$.

Appendix B Additional Results

To check the validity of the regression procedure conducted in a latent space, we compared the latent vector obtained by regression with the latent vector obtained by reconstruction for an input data pair. For the reconstruction, we used both the joint vector and the corresponding image in the H3.6m dataset [20]. For the regression, only the joint vector was given and the projected point was estimated by the proposed regression method. Since the latent vectors were obtained from the same input data, in ideal conditions the vectors should converge to the same location. Figure 11 shows the qualitative results for the regression and the reconstruction for the same input data pair, where it can be seen that both responses converged to the ground truth image. The graph in Figure 13 indicates the KL-divergence between the two latent vectors.

Since the latent vector $z$ in the paper is defined by a Gaussian distribution, we used the KL-divergence as distance measure. As seen in the graph, the KL-divergence obtained by the proposed method was gradually decreased. The result demonstrates that the two vectors obtained from both cases converged to the same location, as expected. When tested with the C-VAE (2), the divergence did not converge.

As shown in Figure 14, we also measured the changes of the negative log likelihood (NLL), for both the regression and the reconstruction. In the proposed method, we confirmed that the NLL ratio for both cases converged. In C-VAE (2), the NLL ratio converged only when both the joint vector and the images were given, but it did not successfully converge when the regression was applied.

Figure 15, Figure 16 and Figure 17 show additional generation results of sports sequences. The figures describe the regression results of the proposed method and of R-VAE in our work, which are the supplementary results of Figure 7 included in the submitted version. We confirmed that the proposed method achieved a superior regression performance for diverse action sequences compared to R-VAE.

In addition to the image sequence regression and human pose reconstruction examples, we newly performed human joint estimation experiments as shown in Fig. 12. Although our method is not originally designed for the human joint estimation purpose, the proposed method could successfully estimate the human joints by performing regression using an (image - joint) data pair and finding the corresponding joints when a new image is given. This example further shows that the proposed method is applicable to practical applications composed of various data pairs.