The Angel is in the Priors: Improving GAN based Image and Sequence Inpainting with Better Noise and Structural Priors

TL;DR

This paper introduces a fully unsupervised GAN-based inpainting framework that uses learned priors for faster inference and higher quality, extending to sequence inpainting with recurrent networks and consistency losses.

Contribution

It presents a data-driven prior prediction network for GAN inpainting, enabling single-pass inference and applying structural and sequence priors for improved results.

Findings

1500X faster inference with learned priors

Higher fidelity in image inpainting

First unsupervised GAN-based sequence inpainting

Abstract

Contemporary deep learning based inpainting algorithms are mainly based on a hybrid dual stage training policy of supervised reconstruction loss followed by an unsupervised adversarial critic loss. However, there is a dearth of literature for a fully unsupervised GAN based inpainting framework. The primary aversion towards the latter genre is due to its prohibitively slow iterative optimization requirement during inference to find a matching noise prior for a masked image. In this paper, we show that priors matter in GAN: we learn a data driven parametric network to predict a matching prior for a given image. This converts an iterative paradigm to a single feed forward inference pipeline with a massive 1500X speedup and simultaneous improvement in reconstruction quality. We show that an additional structural prior imposed on GAN model results in higher fidelity outputs. To extend our model for sequence inpainting, we propose a recurrent net based grouped noise prior learning. To our knowledge, this is the first demonstration of an unsupervised GAN based sequence inpainting. A further improvement in sequence inpainting is achieved with an additional subsequence consistency loss. These contributions improve the spatio-temporal characteristics of reconstructed sequences. Extensive experiments conducted on SVHN, Standford Cars, CelebA and CelebA-HQ image datasets, synthetic sequences and ViDTIMIT video datasets reveal that we consistently improve upon previous unsupervised baseline and also achieve comparable performances(sometimes also better) to hybrid benchmarks.

Tables

Table 1. Table 1 : Comparison of temporal consistency in dB(Eq. 12 )(left section) on synthetic sequences (2000 for CelebA, 4500 for SVHN) by competing algorithms. On right section we also report the mean PSNR of inpainting on different datasets. Higher values of consistency are better. We compare five cases of our proposed framework; M1: Independent Learned noise( z 𝑧 z ) prior, M2: LSTM-CNN grouped learned noise prior, M3: Independent Structural prior, M4: LSTM-CNN grouped noise prior + Structural prior, M5: LSTM-CNN grouped noise prior + Structural prior + Subsequence Consistency Loss. Masks used are RC(Random Central): random 50-70% center mask, RF(Random Freehand): random 50% mask by freehand mask and RCh(Random Checkboard): 50% masked by random checkboard grid masks. In summary : Our unsupervised models have, in general, better temporal consistency and comparable PSNR compared to hybrid(supervised + adversarial) benchmark models.

		Temporal Consistency (dB) on Synthetic Sequences									PSNR (dB) on Single Images
		SVHN @ 64			CelebA @ 128			Cars@ 64			SVHN@ 64			CelebA@ 128			CelebA-HQ @ 256
Genre	Method	RC	RF	RCh	RC	RF	RCh	RC	RF	RCh	RC	RF	RCh	RC	RF	RCh	RC	RF	RCh
	CE[23]	20.5	22.0	21.3	20.1	21.4	19.8	14.2	14.8	13.9	21.9	22.0	21.8	24.3	25.0	24.0	17.8	18.2	17.7
Sup.+	GLCIC[11]	21.6	22.0	21.7	20.9	22.1	20.1	15.9	16.9	15.7	23.2	24.0	22.7	27.9	28.0	27.2	23.8	23.7	22.7
Adv	GIP[30]	21.7	22.9	23.0	21.1	22.2	21.3	16.1	17.2	15.9	23.9	24.1	23.0	28.2	28.7	27.7	24.1	24.3	23.6
	GFC [19]	-	-	-	23.1	24.9	23.8	-	-	-	-	-	-	28.0	28.2	27.1	-	-	-
	Yeh et al.[29]	22.5	22.9	22.8	21.9	22.2	21.1	13.9	14.0	13.2	20.9	21.2	21.0	23.0	23.1	21.4	15.7	16.0	13.1
	Proposed:M1	23.8	25.9	24.2	22.6	24.0	23.0	15.2	15.6	15.1	23.0	23.8	22.5	24.8	25.2	23.7	20.1	20.4	18.9
Unsup	Proposed:M2	25.0	26.9	25.9	23.8	25.6	24.2	-	-	-	-	-	-	-	-	-	-	-	-
	Proposed:M3	-	-	-	24.1	27.4	27.1	-	-	-	-	-	-	27.4	27.9	26.4	22.6	23.0	22.0
	Proposed:M4	-	-	-	26.3	29.8	29.4	-	-	-	-	-	-	-	-	-	-	-	-
	Proposed:M5	-	-	-	27.6	28.0	26.9	-	-	-	-	-	-	-	-	-	-	-	-

Table 2. Table 2 : Comparison of MOVIE metric [ 25 ] averaged over test sequences of VidTIMIT dataset. Lower value of metric is better for perceptual quality of a reconstructed sequence. Refer to Table 1 for definition of proposed methods M1-M5. M6 is framework of M5 but structural priors are evaluated every alternate third frame.

Competing					Proposed
[29]	[23]	[11]	[30]	[19]	M1	M2	M4	M5	M6
0.68	0.60	0.52	0.42	0.35	0.48	0.31	0.23	0.18	0.22

Equations

$$\underset{G_{{\theta }_G}}{\min }\underset{D_{{\theta }_D}}{\max }V(D_{{\theta }_D},G_{{\theta }_G})={\mathbb{E}}_{x\sim p_{data}(x)}[\log D_{{\theta }_D}(x)]$$

$$+{\mathbb{E}}_{z\sim p_z(z)}[1-D_{{\theta }_d}(G_{{\theta }_G}(z))].$$

$$\hat{z}=\underset{z}{\mathrm{argmin}}L(M\odot G(z),M\odot I).$$

$$L_c=M\odot (I-G(P_{{\theta }_z}(I_d))$$

$$L_r=\log (1-D_{{\theta }_d}(G_{{\theta }_G}(P_{{\theta }_z}(I_d)))$$

$$L_g=M\odot |{\nabla }_x{I}_d-{\nabla }_{x}G()|$$

$$+M\odot |{\nabla }_y{I}_d-{\nabla }_{y}G(P_{{\theta }_z}(I_d))|$$

$$L_z^{com}={\lambda }_{1}Lc+{\lambda }_{2}Lr+{\lambda }_3{L}_g,$$

$$\hat{I}=M\odot I+(1-M)\odot G_{{\theta }_G}(P_{{\theta }_z}(I_d)).$$

$$\underset{G_{{\theta }_G}}{\min }\underset{D_{{\theta }_D}}{\max }V(D_{{\theta }_D},G_{{\theta }_G})={\mathbb{E}}_{x,c\sim p_{data}(x,c)}[\log D(x,c)]$$

$$+{\mathbb{E}}_{z\sim p_z(z)}[1-D_{{\theta }_d}(G_{{\theta }_G}(z))].$$

$$L_z^{gr}=\frac{1}{W}\sum _{i=1}^W{L}_z^{com}(I_d^i,I_p^i).$$

$$L_z^{ss}=\frac{1}{\left(\genfrac{}{}{0ex}{}{W}{2}\right)}\sum _{\forall (i,j)\in W}||z_p^i-z_p^j|{|}^2.$$

$$L_z^{LSTM}=L_z^{gr}+{\lambda }_4{L}_z^{ss},$$

$${\eta }_{temp}=\frac{1}{\left(\genfrac{}{}{0ex}{}{N}{2}\right)}\sum _{\forall (i,j)\in S_N}PSNR({\hat{I}}^i,{\hat{I}}^j)$$

Full text

1 Introduction

Image inpainting usually refers to filling up of holes or masked regions with plausible pixel values coherent with the neighborhood context. Traditional techniques [2, 9] were mainly successful in inpainting background and scenes with repetitive textures by matching and copying background patches into holes. However, these methods fail on cases where patterns are unique or non repetitive such as on faces and objects. Also, these methods fail to capture higher semantics of the scene. With the recent breakthrough in generative models such as Variational Autoencoeder (VAE)[14] and Generative Adversarial Networks (GAN) [8], inpainting, in general, is seen as an image completion problem. There are mainly two schools of approach, viz. a) completely unsupervised: conditioned on a prior latent/noise vector [29] b) mixture of supervised + unsupervised: conditioned on masked image [23, 11]. The latter methods heavily depend on an initial phase of fully supervised training (reconstruction loss between original and inpainted outputs within the mask), followed by refinement stage with adversarial loss to add high frequency components in reconstructions. Going against the trend, we feel, the true essence of GAN lies in its ability to generate data within a completely unsupervised framework. The former method of [29] is thus more difficult to train because it has to ‘hallucinate’ an entire object with just a noise/latent vector conditioning and no information of masked/damaged pixels. Thus, though, the latter school of approach has gained major attention among inpainting community, in this paper, we advocate the former genre of unsupervised approach(pixel values under mask never used). Being unsupervised is the merit of [29], but it also creates a run time bottleneck. The algorithm follows iterative gradient descent optimization for finding the ‘best matching’ noise prior corresponding to damaged image. Such iterative framework prohibits real time applications.

In this paper we primarily aim to massively accelarate inference runtime (we achieve 1500X speedup compared to [29]) with simultaneousness visual quality improvement by parametrically learning noise priors. Another issue with inpainting(both supervised and unsupervised) is multi modal completion possibility of a masked region. For example, a masked lip region of face may be completed as smiling or neutral. We show that it is possible to regularize the inpainted outputs with some structural priors. As an example, for a face, we can make use of the facial landmarks as priors. Lastly, single image inpainting models cannot be appreciable applied on videos. Though each frame might be visually pleasing, when viewed as a sequence, there are lot of jitter and flicker due to temporal inconsistency of models. We propose to subdue such inconsistencies with a recurrent net based grouped noise prior learning combined with a subsequence consistency constraint. Our contributions can be summarized as follows:

Unsupervised data driven GAN noise prior prediction framework to convert the iterative paradigm of [29] to a single feed forward pipeline with visually better reconstruction and simultaneous massive speedup of inference time by 1500$\times$. 2. 2.

Augmenting structural priors to improve GAN samples which eventually results in better reconstructions. Such priors also regularize GAN training to respect pose and size of objects. 3. 3.

Pioneering effort towards GAN based sequence inpainting with a recurrent neural net based grouped prior learning for better temporal consistency of reconstructed sequences compared to both supervised and unsupervised benchmarks. 4. 4.

A sub-sequence consistency loss to further improve temporal smoothness of reconstructed sequences 5. 5.

We exhaustively validate our models on CelebA, SVHN, Standford Cars, CelebaHQ image datasets and VidTIMIT video dataset.

2 Related works

Traditional image inpainting methods[1, 4, 6, 7] broadly worked with matching patches and diffusion of low level features from unmasked sections to the masked region. These method mainly worked on synthesis of stationary textures of background scenes where it is plausible to find a matching patch from unmasked regions. However, complex objects lack such redundancy of appearance features and thus recent methods leverage hierarchical feature learning capability of deep neural nets to learn higher order semantics of a scene. Initial deep learning based methods [15, 28] were completely supervised and trained with conservative $\mathcal{L}_{2}$ reconstruction loss. With the advent of GANs, a common practice [23, 11] has been to refine the blurry reconstructions by $\mathcal{L}_{2}$ loss with an adversarial loss coming from a discriminator which is also simultaneously trained to distinguish real samples from inpainted samples. Notably, the first work within this paradigm of approach was Context Encoder(CE) [23] by Pathak et al., in which the authors tried to learn scene representation along with inpainting. Iizuka et al. proposed ‘Globally and Locally Consistent Image Completion’ (GLCIC) in which a inpainter/generator network is pitted against two discriminators, one for gauging realism of entire image and the other for measuring fidelity of local reconstruction of masked patch region. Recently, Yu et al. [30] improved upon GLCIC, by incorporating contextual attention within inpainting network so that the net learns to leverage distant information from uncorrupted pixels. These methods have a common pipeline of fully supervised training stage followed by adversarial loss based refinement. Thus these methods are not fully unsupervised since paired examples(masked and unmasked) are required during training.

In this paper, we are advocating a fully unsupervised approach (information about the masked pixels not used anywhere in training pipeline) to inpainting pioneered by Yeh et al. [29]. In [29], the idea is to first train a GAN framework conditioned on only noise prior($z$) sampled from some prior known distribution. At test time, since their method is completely unsupervised, the authors used an iterative gradient descent optimization to find the ‘best matching’ $z$ vector for the damaged image with the pre-trained generator and discriminator network of the GAN. However, this iterative optimization takes about 2.5 minutes/image and is thus not suitable for practical applications. We consider the framework of [29] as a baseline and seek to improve upon the inference time and reconstruction quality. In the process, we also achieve comparable performance to the contemporary hybrid trained methods.

3 Background

3.1 GAN Basics

Proposed by Goodfellow et al.[8], a GAN model consists of two parametrized deep neural nets, viz., generator, $G_{\theta_{G}}$, and discriminator, $D_{\theta_{D}}$. The task of the generator is to yield an image, $x\in\mathcal{R}^{H\times W\times 3}$ with a latent noise prior vector, $z\in\mathcal{R}^{d}$, as input. $z$ is sampled from a known distribution, $p_{z}(z)$. A common choice [8] is, $z\sim\mathcal{U}[-1,1]^{d}$. The discriminator is pitted against the generator to distinguish real samples(sampled from $p_{data}$) from fake/generated samples. Specifically, discriminator and generator play the following game on $V(D_{\theta_{D}},G_{\theta_{G}})$:

[TABLE]

[TABLE]

With enough capacity, on convergence, $G_{\theta_{G}}$ fools $D_{\theta_{D}}$ at random [8].

3.2 Baseline GAN based unsupervised inpainting

We first review the unsupervised inpainting baseline of Yeh et al. [29]. Given a damaged image, $I_{d}$, corresponding to an original image, $I$, and a pre-trained GAN model, the idea is to iteratively find the ‘closest’ $z$ vector (starting randomly from $\mathcal{U}[-1,1]^{d}$) which results in a reconstructed image whose semantics are similar to corrupted image. $z$ is optimized as,

[TABLE]

where $M$ is the binary mask with zeros on masked region else unity, $\odot$ is the Hadamard operator and $L(\cdot)$ is any loss function. Interesting to note is that the loss function never makes use of pixels inside the masked region. Upon convergence, the inpainted image, $\hat{I}$, is given as, $\hat{I}=M\odot I+(1-M)\ \odot G_{\theta_{G}}(\hat{z})$.

4 Proposed Method

4.1 Data driven Noise Prior Learning

Though the unsupervised characteristic of [29] is encouraging for the generative learning community, the iterative optimization is a major bottleneck in the pipeline. Instead of iteratively optimizing the noise prior, $z$, for each test image during runtime, we propose to learn an unsupervised offline parametric model, $P_{\theta_{z}}$, for predicting $z$ vector. The parameter set, $\theta_{z}$, is optimized to minimize the following unsupervised losses:

Contextual Loss: This loss ensures that the predicted noise prior preserves fidelity with respect to the original unmasked regions.

[TABLE]

Realism Loss: This loss ensures that the inpainted output lies near the original/real data manifold and is measured by the log likelihood of belongingness to real class assigned by the pre-trained discriminator

[TABLE]

Gradient Difference Loss: Inspired by [21, 22] we also use the gradient difference loss imposed between the gradient (horizontal and vertical) matrices of original and reconstructed outputs. This compels the network to predict noise priors which yield high frequency retaining samples and also respects the gradients of the original scene.

[TABLE]

[TABLE]

Please note that the loss is still calculated on the unmasked regions only. In summary, parameter set, $\theta_{z}$, is optimized to minimize the combined loss, $L_{z}^{com}$,

[TABLE]

where $\lambda_{i}$’s controls the relative importance of each loss factor. After convergence of training of $P_{\theta_{z}}$, given a masked image, $I_{d}$, mask, $M$, we can get the inpainted output, $\hat{I}$, in one feed forward step instead of the iterative optimizations of [29]. Inpainted image, $\hat{I}$, is given by,

[TABLE]

Though Eq. 2 and 6 are functionally same, prediction using a learned parametric network tends to perform better than ad hoc iterative optimization. This is because, with evolution of training, the network learns to adapt parameters to map images with closely matching appearances to similar $z$ vectors. Parameter update for a given image thus implicitly generalizes to images with similar characteristics.

4.2 Regularization with Structural Priors

Image inpainting intrinsically suffers from a multi modal completion problem. A given masked region has multiple plausible possibilities for completion. For example, consider Fig.2: for an unconstrained optimization setup, the masked region of the face can be inpainted with different facial expressions. From a single image inpainting point of view this might not be an issue. But in case of sequences, it is desirable to maintain a smooth flow of scene dynamics. A laughing face, for example, cannot suddenly be inpainted as a neutral frame. We propose to further regularize our network by augmenting structural priors. Structural priors can be any representation which captures the pose and size of the object to be inpainted and thereby compelling the network to yield outputs by respecting such priors. Such additional priors can be seen as conditional variables, $c$, to the GAN framework. Formulation of Eq. 1 changes subtly to respect the joint distribution of real samples and conditional information. The modified game, $V(D_{\theta_{D}},G_{\theta_{G}})$:

[TABLE]

[TABLE]

The noise prior predictor network, $P_{\theta_{z}}$ has to optimize $\theta_{z}$ by respecting the structural prior as an additional constraint.

In this paper, without any loss of generalization, we have considered face inpainting with semantic priors as facial landmarks automatically extracted in real time(5ms @ 256$\times$256 resolution) using the robust framework of Kazemi et al. [13] which achieves benchmark performance on face alignment.

4.3 Grouped Noise Prior Learning for Sequences

To our best knowledge, this is the first demonstration of GAN based completely unsupervised sequence inpainting. A naive approach of applying the formulation of Eq. 6 on sequences is to inpaint individual frames independently. However, such anapproach fails to learn the temporal dynamics of sequence and thereby yielding jittering effects. In this regard, for a sequence of $N$ frames, we propose to use a Recurrent Neural Network (RNN) to jointly predict $z$ vectors for a subset of $W$ frames at a time. RNN consist of a hidden state $h_{t}$ to summarize information observed upto that time step. The hidden state is updated after looking at the previous hidden state and the corrupted image(with an additional option to condition on structural priors), leading to more consistent reconstructions in terms of appearance. ames

Since, RNNs suffer from vanishing gradients problem[3] and are unable to capture long dependencies, we use Long Short Term Memory (LSTM) [10] Networks. Fig. 3 shows our LSTM based framework architecture for jointly inpainting a group of frames. Let, $V=\{I_{d}^{1},I_{d}^{2},...,I_{d}^{W}\}$ be a group of $W$ corrupted successive frames. Initially, each frame is passed through a CNN module (same architecture of $P_{\theta_{z}}$, except the last layer outputs $\mathbb{R}^{d\times 1}$ instead of $\mathbb{R}^{1\times 1}$ by $P_{\theta_{z}}$), to obtain the input sequence for the recurrent network $z_{d}^{k}$. We obtain the predicted prior, $z_{p}^{k}$, by feeding the hidden state, $\{h^{k}\}$, of the recurrent network to a fully-connected layer. $z_{p}^{k}$ is then used for reconstructions, $I_{p}^{k}$, with the help of the pre-trained generator, $G_{\theta_{G}}$. We use the loss function in Eq. 6, averaged over the grouped window of $W$ frames to optimize the parameters of LSTM and the CNN descriptor network. Specifically, the grouped prior loss is defined by, $L_{z}^{gr}(\cdot)$,

[TABLE]

Please note, the parameters of pre-trained generator and discriminator are kept frozen. .

4.4 Subsequence consistency loss

We further regularize training of the LSTM framework by an implicit subsequence consistency loss over a group of $W$ neighborhood frames. The motivation is that a group of adjacent frames in a video exhibit close coherence of appearance. Thus, we define a subsequence clique as a collection of $W$ adjacent frames and penalize if the appearances of the frames differ from each other. Disparity between two inpainted images, $I_{p}^{i}$ and $I_{p}^{j}$ can be approximated by Euclidean distance between their latent vectors ($z_{p}^{i},z_{p}^{j}$). We define the loss, $L_{z}^{ss}(\cdot)$ as,

[TABLE]

So, $L_{z}^{gr}$ helps in learning the temporal dynamics while $L_{z}^{ss}$ explicitly fosters temporal smoothness. If $L_{z}^{ss}$ dominates then, the network will be penalized by $L_{z}^{gr}$ because over smoothing of a sequence is not a true characterization of a real world sequence. The final loss function for the LSTM-CNN combined framework is given by $L_{z}^{LSTM}$,

[TABLE]

where $\lambda_{4}$ sets relative importance of subsequence consistency. Please note, $L_{z}^{ss}(\cdot)$ is applied only on a neighborhood of $W$ frames and not on entire sequence. Applying $L_{z}^{ss}(\cdot)$ on entire sequence is not a true representation of temporal dynamics because we will be then penalizing appearance changes even over distant frames. On contrary, reducing $W=1$ means no explicit temporal consistency loss.

5 Results

5.1 Single Image Inpainting

We experiment on cropped SVHN[12], Standford Cars[16], CelebA[20] and CelebA-HQ[20]. SVHN crops are resized to 64$\times$64. On Standford Cars we use bounding box information to extract and resize cars to 64$\times$64. CelebA images are center cropped to 64$\times$64 and 128$\times$128. Celeb-HQ images are resized to 256$\times$256. On SVHN and Cars, we use the dataset provider’s test/train split. For CelebA and CelebA-HQ, we keep 2000 images for testing.

5.1.1 Importance of Learned Noise Prior:

The most important improvement that we achieve over [29] is a significant speedup during inferencing. In Fig. 4 we compare the initial solution of [29] with our one shot feed forward solution. Without any mechanism to estimate noise prior from masked image, initial solutions of [29] lie far from real data manifold and thereby mandating an iterative approach. Abiding by the suggestions in [29], each image requires 1500 test time iterations. Our approach adds just subtle amount of computation for the noise predictor network and a negligible overhead for the structural priors; thereby making our model almost 1500X faster compared to [29]. From Fig. 12, it is encouraging to see that even after the iterative optimization, visual quality of our method is usually superior than [29].

5.1.2 Importance of Structural Priors:

In this paper, we have considered the special case of face inpainting with semantic priors as facial landmarks detected by the robust framework of [13]. We observed three fold benefits of leveraging such priors.

Improved GAN Samples and Reconstructions: Conditioning on structural priors forces the generator to yield samples closer to natural data manifold. Random samples from such conditioned generator are thus more photo-realistic (see Fig. 5) compared to the unconditioned vanilla version of GAN used by [29]. Towards this end, we visually compare (following the protocol in [26]) the quality of random samples from our proposed semantic conditioned GAN and [29] at resolutions of 64$\times$64 and 128$\times$128. For visual turing test, a human annotator is randomly shown total 200 images(100 real and 100 generated) in groups of 20 and asked to label each sample as real or fake. Decisions from 10 annotators are taken. On average, 64$\times 64$ resolution, the classification accuracy is 5.8% higher for DIP($p=10^{-3}$) and 4.2% higher($p=10^{-2}$) at 128$\times$128 resolution. Thus, human annotators found it more difficult to distinguish samples from our model compared to DIP.

Control of Pose and Expression: With structural priors, the generator learns to disentangle appearance and pose. A given semantic prior should force the generator to create a face with matching head pose and facial expression while two nearby $z$ vectors results in similar facial textures. In Figure 6(Top setting) we show such disentanglement learned by our model.

Greater structural fidelity to reference image: In Fig. 8, we show the importance of structural priors on top of learned noise priors. Reconstructions with only our proposed learned noise priors might be stand alone realistic but are not penalized for changing facial expressions. For example, a (masked) smiling face can be inpainted as a neutral face by only conditioning on a learned noise prior. However, if we constrain the model with structural priors, the reconstructions are more coherent in appearance and expression to the reference image. Such structural fidelity is key in achieving temporally more consistent sequence reconstructions as discussed in upcoming sections.

5.1.3 Comparison to Hybrid Benchmarks

Though our method is unsupervised, for completeness of the paper, we also compare with recent hybrid inpainting benchmarks of [23, 11, 30, 19]. To scale up our GAN model to 256$\times$256, we follow the progressive training strategy of [12]. See Fig. 13 for visual examples.

**Is Supervised Phase Mandatory ?

**To seek an answer to this, we trained the models of [23, 11, 30, 19] without any $\mathcal{L}_{2}$ loss but only adversarial loss. We observe that these methods fail to perform in absence of $\mathcal{L}_{2}$ loss. In Fig. 12, we show some visual examples.

5.2 Sequence Inpainting

5.2.1 Temporal consistency and Synthetic Sequences

Recent deep learning based inpainting works have only been restricted for single image inpainting. The genre of video has not received interest. Even if there are some works [18, 27], the reported results are in terms of per frame PSNR which does not take into account the temporal consistency/dynamics of scene reconstructions. For example, it is very annoying for a viewer if the stationary portions of a series of frames are reconstructed with different appearances on each frame and thereby creating jitter effects.

We dedicate this section to analyze the temporal consistencies of different methods on synthetic sequences. A synthetic sequence, $S_{T}$, of length $T$ is formed by taking a single image, $I$, and masking it with $T$ different/same corruptions masks. An ideal inpainting model should be agnostic of the corruption masks and yield identical reconstructions for all the $n$ frames. We define temporal consistency, $\eta_{temp}$, as the mean pairwise PSNR between all possible pairs($\hat{I}^{i},\hat{I}^{j}$) of inpainted frames within a synthetic sequence, $S_{N}$, of length, $N$;

[TABLE]

Eq. 12 allows enumerating the consistency of a generative model. Ideally, we want $\eta_{temp}$=0. Please note that this evaluation is not possible on real videos because the transformation from one frame to another is not known and thus it is not possible to align the frames to a single frame of reference without incorporating interpolation noise with motion compensator[5]. In Table 1 we compare the consistencies with contemporary benchmarks. We see progressive improvement of consistency with the addition of LSTM-grouped prior and structural priors. Note, even the hybrid (supervised + adversarial) benchmarks manifests higher inconsistencies with exception of [19] because it jointly trains the network with inpainting loss and face parsing loss. This bolsters the hypothesis that a prior knowledge of object structure helps in inpainting.

5.2.2 Importance of Subsequence Consistency Loss

In Fig. 10 we show a synthetic sequence, in which a same face is masked differently. Proposed LSTM-grouped prior based reconstruction is successful is maintaining the overall same facial expression but fails to maintain subtle textural consistencies as shown in highlighted insets. Subsequence consistency loss helps in maintaining such subtle texture coherence which results in improved temporal consistency. Again, please note, these difference are much more easier to illustrate(and visualize) in such synthetic sequences than on real videos.

5.2.3 Application on Real Videos:

The experiments with synthetic sequences taught us three lessons, viz., a)LSTM-CNN based grouped noise prior learning is better than independent noise prior learning b) structural prior fosters in higher fidelity and c) subsequence consistency loss helps in preserving subtle texture details. With these knowledge, we proceed to demonstrate first attempt towards GAN based inpainting on real videos. For this, we selected the VidTIMIT dataset[24] which consists of video recordings of 43 subjects each narrating 10 different sentences. Images of CelebA dataset are of superior resolution than those of VidTIMIT. Due to this intrinsic difference of data distribution we finetuned our pretrained(trained on CelebA) models on randomly selected 33 subjects of VidTIMIT. Remaining videos of 10 subjects were kept for testing inpainting performances. In total, there are total 9600 frames for testing. All faces center cropped to 128$\times$128.

Evaluating Video Quality: MOVIE metric [25]: Traditional metrics such as PSNR and structure similarity (SSIM) are not a true reflection of human visual perception measure as shown in recent studies [30, 17]. Also, these metrics donot consider any temporal information. For this, we preferred to use the MOVIE metric [25]. MOVIE is a spatio-spectrally localized framework for assessing a video quality by considering spatial, temporal and spatio-temporal aspects. A lower value of MOVIE metric indicates a better video. MOVIE metric was found to appreciably correlate with human perception. In Table 2, we compare the average test set MOVIE metric. All variants of our proposed framework outperforms [29]. With independent noise prior model, we get better performance than [23] and comparable performance to [11, 30]. Addition of LSTM-grouped prior and structural prior boosts our performance with further improvement coming from subsequence consistency loss. It is interesting to see that even if we compute structural prior on third(and reuse in between), there is subtle degradation of performance. We show some video snippets in Fig. 11.

6 Discussion and Conclusion

In this paper, we showed the importance of priors in GANs for pushing the performance envelope of unsupervised inpainting framework of [29] with better inpainting quality and almost 1500$\times$ speedup. The objective of this paper was to purposefully abstain from the contemporary practice of hybrid(supervised + unsupervised) training and focus on creating a faster unsupervised framework comparable visual performance. Our proposed framework with grouped LSTM-CNN guided noise prior and structural prior manifests better spatio-temporal characteristics than contemporary hybrid baselines. This shows that current single image inpainting methods have further scopes of improvement on videos and the frameworks used by us in this regard can be exploited by those algorithms also. Given the current state of GAN research, it is not expected that a completely unsupervised GAN based inpainter can work on natural images such as ImageNet or Places2 dataset(which hybrid methods are capable of due to supervised $\mathcal{L}_{2}$). However, as our understanding on GANs improve and we enable GAN models to generate natural scenes, the methods of this paper shall seamlessly fit in those scenarios as well.